v2.4.3

Prepare changelog for v2.4.3
v2.4.3-rc.1
2019-11-04 13:51:50 +00:00 · 2019-11-04 13:51:49 +00:00 · 2019-10-30 16:40:25 +00:00 · 2019-10-30 16:40:25 +00:00 · 2019-10-26 13:45:24 -04:00 · 2019-10-25 23:24:30 -04:00
37 changed files with 2244 additions and 649 deletions
@@ -1,3 +1,171 @@
+Changes in [2.4.3](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.3) (2019-11-04)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.4.3-rc.1...v2.4.3)
+
+ * No changes since rc.1
+
+Changes in [2.4.3-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.3-rc.1) (2019-10-30)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.4.2...v2.4.3-rc.1)
+
+ * fix the path in references to logger.js
+   [\#1056](https://github.com/matrix-org/matrix-js-sdk/pull/1056)
+ * verification in DMs
+   [\#1050](https://github.com/matrix-org/matrix-js-sdk/pull/1050)
+ * Properly documented the function possible returns
+   [\#1054](https://github.com/matrix-org/matrix-js-sdk/pull/1054)
+ * Downgrade to Bluebird 3.5.5 to fix Firefox
+   [\#1055](https://github.com/matrix-org/matrix-js-sdk/pull/1055)
+ * Upgrade safe deps to latest major version
+   [\#1053](https://github.com/matrix-org/matrix-js-sdk/pull/1053)
+ * Don't include .js in the import string.
+   [\#1052](https://github.com/matrix-org/matrix-js-sdk/pull/1052)
+
+Changes in [2.4.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.2) (2019-10-18)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.4.2-rc.1...v2.4.2)
+
+ * No changes since v2.4.2-rc.1
+
+Changes in [2.4.2-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.2-rc.1) (2019-10-09)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.4.1...v2.4.2-rc.1)
+
+ * Log state of Olm sessions
+   [\#1047](https://github.com/matrix-org/matrix-js-sdk/pull/1047)
+ * Add method to get access to all timelines
+   [\#1048](https://github.com/matrix-org/matrix-js-sdk/pull/1048)
+
+Changes in [2.4.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.1) (2019-10-01)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.4.0...v2.4.1)
+
+ * Upgrade deps
+   [\#1046](https://github.com/matrix-org/matrix-js-sdk/pull/1046)
+ * Ignore crypto events with no content
+   [\#1043](https://github.com/matrix-org/matrix-js-sdk/pull/1043)
+
+Changes in [2.4.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.0) (2019-09-27)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.4.0-rc.1...v2.4.0)
+
+ * Clean Yarn cache during release
+   [\#1045](https://github.com/matrix-org/matrix-js-sdk/pull/1045)
+
+Changes in [2.4.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.4.0-rc.1) (2019-09-25)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.3.2...v2.4.0-rc.1)
+
+ * Remove id_server from creds for interactive auth
+   [\#1044](https://github.com/matrix-org/matrix-js-sdk/pull/1044)
+ * Remove IS details from requestToken to HS
+   [\#1041](https://github.com/matrix-org/matrix-js-sdk/pull/1041)
+ * Add support for sending MSISDN tokens to alternate URLs
+   [\#1040](https://github.com/matrix-org/matrix-js-sdk/pull/1040)
+ * Add separate 3PID add and bind APIs
+   [\#1038](https://github.com/matrix-org/matrix-js-sdk/pull/1038)
+ * Bump eslint-utils from 1.4.0 to 1.4.2
+   [\#1037](https://github.com/matrix-org/matrix-js-sdk/pull/1037)
+ * Handle WebRTC security errors as non-fatal
+   [\#1036](https://github.com/matrix-org/matrix-js-sdk/pull/1036)
+ * Check for r0.6.0 support in addition to unstable feature flags
+   [\#1035](https://github.com/matrix-org/matrix-js-sdk/pull/1035)
+ * Update room members on member event redaction
+   [\#1030](https://github.com/matrix-org/matrix-js-sdk/pull/1030)
+ * Support hidden read receipts
+   [\#1028](https://github.com/matrix-org/matrix-js-sdk/pull/1028)
+ * Do 3pid lookups in lowercase
+   [\#1029](https://github.com/matrix-org/matrix-js-sdk/pull/1029)
+ * Add Synapse admin functions for deactivating a user
+   [\#1027](https://github.com/matrix-org/matrix-js-sdk/pull/1027)
+ * Fix addPendingEvent with pending event order == chronological
+   [\#1026](https://github.com/matrix-org/matrix-js-sdk/pull/1026)
+ * Add AutoDiscovery.getRawClientConfig() for easy .well-known lookups
+   [\#1024](https://github.com/matrix-org/matrix-js-sdk/pull/1024)
+ * Don't convert errors to JSON if they are JSON already
+   [\#1025](https://github.com/matrix-org/matrix-js-sdk/pull/1025)
+ * Send id_access_token to HS for use in proxied IS requests
+   [\#1022](https://github.com/matrix-org/matrix-js-sdk/pull/1022)
+ * Clean up JSON handling in identity server requests
+   [\#1023](https://github.com/matrix-org/matrix-js-sdk/pull/1023)
+ * Use the v2 (hashed) lookup for identity server queries
+   [\#1021](https://github.com/matrix-org/matrix-js-sdk/pull/1021)
+ * Add getIdServer() & doesServerRequireIdServerParam()
+   [\#1018](https://github.com/matrix-org/matrix-js-sdk/pull/1018)
+ * Make requestToken endpoints work without ID Server
+   [\#1019](https://github.com/matrix-org/matrix-js-sdk/pull/1019)
+ * Fix setIdentityServer
+   [\#1016](https://github.com/matrix-org/matrix-js-sdk/pull/1016)
+ * Change ICE fallback server and make fallback opt-in
+   [\#1015](https://github.com/matrix-org/matrix-js-sdk/pull/1015)
+ * Throw an exception if trying to do an ID server request with no ID server
+   [\#1014](https://github.com/matrix-org/matrix-js-sdk/pull/1014)
+ * Add setIdentityServerUrl
+   [\#1013](https://github.com/matrix-org/matrix-js-sdk/pull/1013)
+ * Add matrix base API to report an event
+   [\#1011](https://github.com/matrix-org/matrix-js-sdk/pull/1011)
+ * Fix POST body for v2 IS requests
+   [\#1010](https://github.com/matrix-org/matrix-js-sdk/pull/1010)
+ * Add API for bulk lookup on the Identity Server
+   [\#1009](https://github.com/matrix-org/matrix-js-sdk/pull/1009)
+ * Remove deprecated authedRequestWithPrefix and requestWithPrefix
+   [\#1000](https://github.com/matrix-org/matrix-js-sdk/pull/1000)
+ * Add API for checking IS account info
+   [\#1007](https://github.com/matrix-org/matrix-js-sdk/pull/1007)
+ * Support rewriting push rules when our internal defaults change
+   [\#1006](https://github.com/matrix-org/matrix-js-sdk/pull/1006)
+ * Upgrade dependencies
+   [\#1005](https://github.com/matrix-org/matrix-js-sdk/pull/1005)
+
+Changes in [2.3.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.3.2) (2019-09-16)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.3.2-rc.1...v2.3.2)
+
+ * [Release] Fix addPendingEvent with pending event order == chronological
+   [\#1034](https://github.com/matrix-org/matrix-js-sdk/pull/1034)
+
+Changes in [2.3.2-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.3.2-rc.1) (2019-09-13)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.3.1...v2.3.2-rc.1)
+
+ * Synapse admin functions to release
+   [\#1033](https://github.com/matrix-org/matrix-js-sdk/pull/1033)
+ * [To Release] Add matrix base API to report an event
+   [\#1032](https://github.com/matrix-org/matrix-js-sdk/pull/1032)
+
+Changes in [2.3.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.3.1) (2019-09-12)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.3.1-rc.1...v2.3.1)
+
+ * No changes since rc.1
+
+Changes in [2.3.1-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.3.1-rc.1) (2019-09-11)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.3.0...v2.3.1-rc.1)
+
+ * Update room members on member event redaction
+   [\#1031](https://github.com/matrix-org/matrix-js-sdk/pull/1031)
+
+Changes in [2.3.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.3.0) (2019-08-05)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.3.0-rc.1...v2.3.0)
+
+ * [release] Support rewriting push rules when our internal defaults change
+   [\#1008](https://github.com/matrix-org/matrix-js-sdk/pull/1008)
+
+Changes in [2.3.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.3.0-rc.1) (2019-07-31)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.2.0...v2.3.0-rc.1)
+
+ * Add support for IS v2 API with authentication
+   [\#1002](https://github.com/matrix-org/matrix-js-sdk/pull/1002)
+ * Tombstone bugfixes
+   [\#1001](https://github.com/matrix-org/matrix-js-sdk/pull/1001)
+ * Support for MSC2140 (terms of service for IS/IM)
+   [\#988](https://github.com/matrix-org/matrix-js-sdk/pull/988)
+ * Add a request method to /devices
+   [\#994](https://github.com/matrix-org/matrix-js-sdk/pull/994)
+
 Changes in [2.2.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v2.2.0) (2019-07-18)
 ================================================================================================
 [Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v2.2.0-rc.2...v2.2.0)
@@ -322,13 +322,13 @@ To provide the Olm library in a browser application:
 
 To provide the Olm library in a node.js application:

- * ``yarn add https://packages.matrix.org/npm/olm/olm-3.0.0.tgz``
+ * ``yarn add https://packages.matrix.org/npm/olm/olm-3.1.4.tgz``
   (replace the URL with the latest version you want to use from
    https://packages.matrix.org/npm/olm/)
 * ``global.Olm = require('olm');`` *before* loading ``matrix-js-sdk``.

 If you want to package Olm as dependency for your node.js application, you can
-use ``yarn add https://packages.matrix.org/npm/olm/olm-3.0.0.tgz``. If your
+use ``yarn add https://packages.matrix.org/npm/olm/olm-3.1.4.tgz``. If your
 application also works without e2e crypto enabled, add ``--optional`` to mark it
 as an optional dependency.

@@ -1,6 +1,6 @@
 {
  "name": "matrix-js-sdk",
-  "version": "2.2.0",
+  "version": "2.4.3",
  "description": "Matrix Client-Server SDK for Javascript",
  "main": "index.js",
  "scripts": {
@@ -17,7 +17,7 @@
    "build": "babel -s -d lib src && rimraf dist && mkdir dist && browserify -d browser-index.js | exorcist dist/browser-matrix.js.map > dist/browser-matrix.js && terser -c -m -o dist/browser-matrix.min.js --source-map \"content='dist/browser-matrix.js.map'\" dist/browser-matrix.js",
    "dist": "yarn build",
    "watch": "watchify -d browser-index.js -o 'exorcist dist/browser-matrix.js.map > dist/browser-matrix.js' -v",
-    "lint": "eslint --max-warnings 101 src spec",
+    "lint": "eslint --max-warnings 93 src spec",
    "prepare": "yarn clean && yarn build && git rev-parse HEAD > git-revision.txt"
  },
  "repository": {
@@ -54,11 +54,11 @@
  "dependencies": {
    "another-json": "^0.2.0",
    "babel-runtime": "^6.26.0",
-    "bluebird": "^3.5.0",
+    "bluebird": "3.5.5",
    "browser-request": "^0.3.3",
    "bs58": "^4.0.1",
    "content-type": "^1.0.2",
-    "loglevel": "1.6.1",
+    "loglevel": "^1.6.4",
    "qs": "^6.5.2",
    "request": "^2.88.0",
    "unhomoglyph": "^1.0.2"
@@ -75,19 +75,19 @@
    "eslint": "^5.12.0",
    "eslint-config-google": "^0.7.1",
    "eslint-plugin-babel": "^5.3.0",
-    "exorcist": "^0.4.0",
+    "exorcist": "^1.0.1",
    "expect": "^1.20.2",
    "istanbul": "^0.4.5",
    "jsdoc": "^3.5.5",
    "lolex": "^1.5.2",
    "matrix-mock-request": "^1.2.3",
-    "mocha": "^5.2.0",
+    "mocha": "^6.2.1",
    "mocha-jenkins-reporter": "^0.4.0",
-    "olm": "https://packages.matrix.org/npm/olm/olm-3.1.0.tgz",
-    "rimraf": "^2.5.4",
-    "source-map-support": "^0.4.11",
-    "sourceify": "^0.1.0",
-    "terser": "^4.0.0",
+    "olm": "https://packages.matrix.org/npm/olm/olm-3.1.4.tgz",
+    "rimraf": "^3.0.0",
+    "source-map-support": "^0.5.13",
+    "sourceify": "^1.0.0",
+    "terser": "^4.3.8",
    "watchify": "^3.11.1"
  },
  "browserify": {
@@ -195,6 +195,11 @@ if [ $dodist -eq 0 ]; then
    pushd "$builddir"
    git clone "$projdir" .
    git checkout "$rel_branch"
+    # We use Git branch / commit dependencies for some packages, and Yarn seems
+    # to have a hard time getting that right. See also
+    # https://github.com/yarnpkg/yarn/issues/4734. As a workaround, we clean the
+    # global cache here to ensure we get the right thing.
+    yarn cache clean
    yarn install
    # We haven't tagged yet, so tell the dist script what version
    # it's building
@@ -273,4 +273,127 @@ describe("SAS verification", function() {
        expect(bob.setDeviceVerified)
            .toNotHaveBeenCalled();
    });
+
+    describe("verification in DM", function() {
+        let alice;
+        let bob;
+        let aliceSasEvent;
+        let bobSasEvent;
+        let aliceVerifier;
+        let bobPromise;
+
+        beforeEach(async function() {
+            [alice, bob] = await makeTestClients(
+                [
+                    {userId: "@alice:example.com", deviceId: "Osborne2"},
+                    {userId: "@bob:example.com", deviceId: "Dynabook"},
+                ],
+                {
+                    verificationMethods: [verificationMethods.SAS],
+                },
+            );
+
+            alice.setDeviceVerified = expect.createSpy();
+            alice.getDeviceEd25519Key = () => {
+                return "alice+base64+ed25519+key";
+            };
+            alice.getStoredDevice = () => {
+                return DeviceInfo.fromStorage(
+                    {
+                        keys: {
+                            "ed25519:Dynabook": "bob+base64+ed25519+key",
+                        },
+                    },
+                    "Dynabook",
+                );
+            };
+            alice.downloadKeys = () => {
+                return Promise.resolve();
+            };
+
+            bob.setDeviceVerified = expect.createSpy();
+            bob.getStoredDevice = () => {
+                return DeviceInfo.fromStorage(
+                    {
+                        keys: {
+                            "ed25519:Osborne2": "alice+base64+ed25519+key",
+                        },
+                    },
+                    "Osborne2",
+                );
+            };
+            bob.getDeviceEd25519Key = () => {
+                return "bob+base64+ed25519+key";
+            };
+            bob.downloadKeys = () => {
+                return Promise.resolve();
+            };
+
+            aliceSasEvent = null;
+            bobSasEvent = null;
+
+            bobPromise = new Promise((resolve, reject) => {
+                bob.on("event", async (event) => {
+                    const content = event.getContent();
+                    if (event.getType() === "m.room.message"
+                        && content.msgtype === "m.key.verification.request") {
+                        expect(content.methods).toInclude(SAS.NAME);
+                        expect(content.to).toBe(bob.getUserId());
+                        const verifier = bob.acceptVerificationDM(event, SAS.NAME);
+                        verifier.on("show_sas", (e) => {
+                            if (!e.sas.emoji || !e.sas.decimal) {
+                                e.cancel();
+                            } else if (!aliceSasEvent) {
+                                bobSasEvent = e;
+                            } else {
+                                try {
+                                    expect(e.sas).toEqual(aliceSasEvent.sas);
+                                    e.confirm();
+                                    aliceSasEvent.confirm();
+                                } catch (error) {
+                                    e.mismatch();
+                                    aliceSasEvent.mismatch();
+                                }
+                            }
+                        });
+                        await verifier.verify();
+                        resolve();
+                    }
+                });
+            });
+
+            aliceVerifier = await alice.requestVerificationDM(
+                bob.getUserId(), "!room_id", [verificationMethods.SAS],
+            );
+            aliceVerifier.on("show_sas", (e) => {
+                if (!e.sas.emoji || !e.sas.decimal) {
+                    e.cancel();
+                } else if (!bobSasEvent) {
+                    aliceSasEvent = e;
+                } else {
+                    try {
+                        expect(e.sas).toEqual(bobSasEvent.sas);
+                        e.confirm();
+                        bobSasEvent.confirm();
+                    } catch (error) {
+                        e.mismatch();
+                        bobSasEvent.mismatch();
+                    }
+                }
+            });
+        });
+
+        it("should verify a key", async function() {
+            await Promise.all([
+                aliceVerifier.verify(),
+                bobPromise,
+            ]);
+
+            // make sure Alice and Bob verified each other
+            expect(alice.setDeviceVerified)
+                .toHaveBeenCalledWith(bob.getUserId(), bob.deviceId);
+            expect(bob.setDeviceVerified)
+                .toHaveBeenCalledWith(alice.getUserId(), alice.deviceId);
+        });
+    });
 });
@@ -43,6 +43,25 @@ export async function makeTestClients(userInfos, options) {
            }
        }
    };
+    const sendEvent = function(room, type, content) {
+        // make up a unique ID as the event ID
+        const eventId = "$" + this.makeTxnId(); // eslint-disable-line babel/no-invalid-this
+        const event = new MatrixEvent({
+            sender: this.getUserId(), // eslint-disable-line babel/no-invalid-this
+            type: type,
+            content: content,
+            room_id: room,
+            event_id: eventId,
+        });
+        for (const client of clients) {
+            setTimeout(
+                () => client.emit("event", event),
+                0,
+            );
+        }
+
+        return {event_id: eventId};
+    };

    for (const userInfo of userInfos) {
        const client = (new TestClient(
@@ -54,6 +73,7 @@ export async function makeTestClients(userInfos, options) {
        }
        clientMap[userInfo.userId][userInfo.deviceId] = client;
        client.sendToDevice = sendToDevice;
+        client.sendEvent = sendEvent;
        clients.push(client);
    }

@@ -154,12 +154,9 @@ describe("MatrixClient", function() {
        });
        // FIXME: We shouldn't be yanking _http like this.
        client._http = [
-            "authedRequest", "authedRequestWithPrefix", "getContentUri",
-            "request", "requestWithPrefix", "uploadContent",
+            "authedRequest", "getContentUri", "request", "uploadContent",
        ].reduce((r, k) => { r[k] = expect.createSpy(); return r; }, {});
        client._http.authedRequest.andCall(httpReq);
-        client._http.authedRequestWithPrefix.andCall(httpReq);
-        client._http.requestWithPrefix.andCall(httpReq);
        client._http.request.andCall(httpReq);

        // set reasonable working defaults
@@ -181,9 +178,6 @@ describe("MatrixClient", function() {
        client._http.authedRequest.andCall(function() {
            return Promise.defer().promise;
        });
-        client._http.authedRequestWithPrefix.andCall(function() {
-            return Promise.defer().promise;
-        });
    });

    it("should not POST /filter if a matching filter already exists", async function() {
@@ -429,6 +429,26 @@ export class AutoDiscovery {
        return AutoDiscovery.fromDiscoveryConfig(wellknown.raw);
    }

+    /**
+     * Gets the raw discovery client configuration for the given domain name.
+     * Should only be used if there's no validation to be done on the resulting
+     * object, otherwise use findClientConfig().
+     * @param {string} domain The domain to get the client config for.
+     * @returns {Promise<object>} Resolves to the domain's client config. Can
+     * be an empty object.
+     */
+    static async getRawClientConfig(domain) {
+        if (!domain || typeof(domain) !== "string" || domain.length === 0) {
+            throw new Error("'domain' must be a string of non-zero length");
+        }
+
+        const response = await this._fetchWellKnownObject(
+            `https://${domain}/.well-known/matrix/client`,
+        );
+        if (!response) return {};
+        return response.raw || {};
+    }
+
    /**
     * Sanitizes a given URL to ensure it is either an HTTP or HTTP URL and
     * is suitable for the requirements laid out by .well-known auto discovery.
@@ -1,6 +1,8 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
 Copyright 2017 Vector Creations Ltd
+Copyright 2019 The Matrix.org Foundation C.I.C.
+Copyright 2019 Michael Telatynski <7t3chguy@gmail.com>

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -23,8 +25,23 @@ limitations under the License.
 * @module base-apis
 */

+import { SERVICE_TYPES } from './service-types';
+import logger from './logger';
+
 const httpApi = require("./http-api");
 const utils = require("./utils");
+const PushProcessor = require("./pushprocessor");
+
+function termsUrlForService(serviceType, baseUrl) {
+    switch (serviceType) {
+        case SERVICE_TYPES.IS:
+            return baseUrl + httpApi.PREFIX_IDENTITY_V2 + '/terms';
+        case SERVICE_TYPES.IM:
+            return baseUrl + '/_matrix/integrations/v1/terms';
+        default:
+            throw new Error('Unsupported service type');
+    }
+}

 /**
 * Low-level wrappers for the Matrix APIs
@@ -46,6 +63,15 @@ const utils = require("./utils");
 *
 * @param {string} opts.accessToken The access_token for this user.
 *
+ * @param {IdentityServerProvider} [opts.identityServer]
+ * Optional. A provider object with one function `getAccessToken`, which is a
+ * callback that returns a Promise<String> of an identity access token to supply
+ * with identity requests. If the object is unset, no access token will be
+ * supplied.
+ * See also https://github.com/vector-im/riot-web/issues/10615 which seeks to
+ * replace the previous approach of manual access tokens params with this
+ * callback throughout the SDK.
+ *
 * @param {Number=} opts.localTimeoutMs Optional. The default maximum amount of
 * time to wait before timing out HTTP requests. If not specified, there is no
 * timeout.
@@ -62,6 +88,7 @@ function MatrixBaseApis(opts) {

    this.baseUrl = opts.baseUrl;
    this.idBaseUrl = opts.idBaseUrl;
+    this.identityServer = opts.identityServer;

    const httpOpts = {
        baseUrl: opts.baseUrl,
@@ -100,6 +127,15 @@ MatrixBaseApis.prototype.getIdentityServerUrl = function(stripProto=false) {
    return this.idBaseUrl;
 };

+/**
+ * Set the Identity Server URL of this client
+ * @param {string} url New Identity Server URL
+ */
+MatrixBaseApis.prototype.setIdentityServerUrl = function(url) {
+    this.idBaseUrl = utils.ensureNoTrailingSlash(url);
+    this._http.setIdBaseUrl(this.idBaseUrl);
+};
+
 /**
 * Get the access token associated with this account.
 * @return {?String} The access_token or null
@@ -461,8 +497,10 @@ MatrixBaseApis.prototype.fetchRelations =
            $relationType: relationType,
            $eventType: eventType,
        });
-    const response = await this._http.authedRequestWithPrefix(
-        undefined, "GET", path, null, null, httpApi.PREFIX_UNSTABLE,
+    const response = await this._http.authedRequest(
+        undefined, "GET", path, null, null, {
+            prefix: httpApi.PREFIX_UNSTABLE,
+        },
    );
    return response;
 };
@@ -955,10 +993,13 @@ MatrixBaseApis.prototype.roomInitialSync = function(roomId, limit, callback) {
 * @param {string} rrEventId ID of the event tracked by the read receipt. This is here
 * for convenience because the RR and the RM are commonly updated at the same time as
 * each other. Optional.
+ * @param {object} opts Options for the read markers.
+ * @param {object} opts.hidden True to hide the read receipt from other users. <b>This
+ * property is currently unstable and may change in the future.</b>
 * @return {module:client.Promise} Resolves: the empty object, {}.
 */
 MatrixBaseApis.prototype.setRoomReadMarkersHttpRequest =
-                                function(roomId, rmEventId, rrEventId) {
+                                function(roomId, rmEventId, rrEventId, opts) {
    const path = utils.encodeUri("/rooms/$roomId/read_markers", {
        $roomId: roomId,
    });
@@ -966,6 +1007,7 @@ MatrixBaseApis.prototype.setRoomReadMarkersHttpRequest =
    const content = {
        "m.fully_read": rmEventId,
        "m.read": rrEventId,
+        "m.hidden": Boolean(opts ? opts.hidden : false),
    };

    return this._http.authedRequest(
@@ -1297,10 +1339,16 @@ MatrixBaseApis.prototype.getThreePids = function(callback) {
 };

 /**
+ * Add a 3PID to your homeserver account and optionally bind it to an identity
+ * server as well. An identity server is required as part of the `creds` object.
+ *
+ * This API is deprecated, and you should instead use `addThreePidOnly`
+ * for homeservers that support it.
+ *
 * @param {Object} creds
 * @param {boolean} bind
 * @param {module:client.callback} callback Optional.
- * @return {module:client.Promise} Resolves: TODO
+ * @return {module:client.Promise} Resolves: on success
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
 MatrixBaseApis.prototype.addThreePid = function(creds, bind, callback) {
@@ -1314,6 +1362,75 @@ MatrixBaseApis.prototype.addThreePid = function(creds, bind, callback) {
    );
 };

+/**
+ * Add a 3PID to your homeserver account. This API does not use an identity
+ * server, as the homeserver is expected to handle 3PID ownership validation.
+ *
+ * You can check whether a homeserver supports this API via
+ * `doesServerSupportSeparateAddAndBind`.
+ *
+ * @param {Object} data A object with 3PID validation data from having called
+ * `account/3pid/<medium>/requestToken` on the homeserver.
+ * @return {module:client.Promise} Resolves: on success
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.addThreePidOnly = function(data) {
+    const path = "/account/3pid/add";
+    return this._http.authedRequest(
+        undefined, "POST", path, null, data, {
+            prefix: httpApi.PREFIX_UNSTABLE,
+        },
+    );
+};
+
+/**
+ * Bind a 3PID for discovery onto an identity server via the homeserver. The
+ * identity server handles 3PID ownership validation and the homeserver records
+ * the new binding to track where all 3PIDs for the account are bound.
+ *
+ * You can check whether a homeserver supports this API via
+ * `doesServerSupportSeparateAddAndBind`.
+ *
+ * @param {Object} data A object with 3PID validation data from having called
+ * `validate/<medium>/requestToken` on the identity server. It should also
+ * contain `id_server` and `id_access_token` fields as well.
+ * @return {module:client.Promise} Resolves: on success
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.bindThreePid = function(data) {
+    const path = "/account/3pid/bind";
+    return this._http.authedRequest(
+        undefined, "POST", path, null, data, {
+            prefix: httpApi.PREFIX_UNSTABLE,
+        },
+    );
+};
+
+/**
+ * Unbind a 3PID for discovery on an identity server via the homeserver. The
+ * homeserver removes its record of the binding to keep an updated record of
+ * where all 3PIDs for the account are bound.
+ *
+ * @param {string} medium The threepid medium (eg. 'email')
+ * @param {string} address The threepid address (eg. 'bob@example.com')
+ *        this must be as returned by getThreePids.
+ * @return {module:client.Promise} Resolves: on success
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.unbindThreePid = function(medium, address) {
+    const path = "/account/3pid/unbind";
+    const data = {
+        medium,
+        address,
+        id_server: this.getIdentityServerUrl(true),
+    };
+    return this._http.authedRequest(
+        undefined, "POST", path, null, data, {
+            prefix: httpApi.PREFIX_UNSTABLE,
+        },
+    );
+};
+
 /**
 * @param {string} medium The threepid medium (eg. 'email')
 * @param {string} address The threepid address (eg. 'bob@example.com')
@@ -1462,7 +1579,9 @@ MatrixBaseApis.prototype.setPusher = function(pusher, callback) {
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
 MatrixBaseApis.prototype.getPushRules = function(callback) {
-    return this._http.authedRequest(callback, "GET", "/pushrules/");
+    return this._http.authedRequest(callback, "GET", "/pushrules/").then(rules => {
+        return PushProcessor.rewriteDefaultRules(rules);
+    });
 };

 /**
@@ -1688,10 +1807,36 @@ MatrixBaseApis.prototype.getKeyChanges = function(oldToken, newToken) {
 // ==========================

 /**
- * Requests an email verification token directly from an Identity Server.
+ * Register with an Identity Server using the OpenID token from the user's
+ * Homeserver, which can be retrieved via
+ * {@link module:client~MatrixClient#getOpenIdToken}.
 *
- * Note that the Home Server offers APIs to proxy this API for specific
- * situations, allowing for better feedback to the user.
+ * Note that the `/account/register` endpoint (as well as IS authentication in
+ * general) was added as part of the v2 API version.
+ *
+ * @param {object} hsOpenIdToken
+ * @return {module:client.Promise} Resolves: with object containing an Identity
+ * Server access token.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.registerWithIdentityServer = function(hsOpenIdToken) {
+    if (!this.idBaseUrl) {
+        throw new Error("No Identity Server base URL set");
+    }
+
+    const uri = this.idBaseUrl + httpApi.PREFIX_IDENTITY_V2 + "/account/register";
+    return this._http.requestOtherUrl(
+        undefined, "POST", uri,
+        null, hsOpenIdToken,
+    );
+};
+
+/**
+ * Requests an email verification token directly from an identity server.
+ *
+ * This API is used as part of binding an email for discovery on an identity
+ * server. The validation data that results should be passed to the
+ * `bindThreePid` method to complete the binding process.
 *
 * @param {string} email The email address to request a token for
 * @param {string} clientSecret A secret binary string generated by the client.
@@ -1703,26 +1848,122 @@ MatrixBaseApis.prototype.getKeyChanges = function(oldToken, newToken) {
 * @param {string} nextLink Optional If specified, the client will be redirected
 *                 to this link after validation.
 * @param {module:client.callback} callback Optional.
+ * @param {string} identityAccessToken The `access_token` field of the identity
+ * server `/account/register` response (see {@link registerWithIdentityServer}).
+ *
 * @return {module:client.Promise} Resolves: TODO
 * @return {module:http-api.MatrixError} Rejects: with an error response.
- * @throws Error if No ID server is set
+ * @throws Error if no identity server is set
 */
-MatrixBaseApis.prototype.requestEmailToken = function(email, clientSecret,
-                                                    sendAttempt, nextLink, callback) {
+MatrixBaseApis.prototype.requestEmailToken = async function(
+    email,
+    clientSecret,
+    sendAttempt,
+    nextLink,
+    callback,
+    identityAccessToken,
+) {
    const params = {
        client_secret: clientSecret,
        email: email,
        send_attempt: sendAttempt,
        next_link: nextLink,
    };
-    return this._http.idServerRequest(
-        callback, "POST", "/validate/email/requestToken",
-        params, httpApi.PREFIX_IDENTITY_V1,
-    );
+
+    try {
+        const response = await this._http.idServerRequest(
+            undefined, "POST", "/validate/email/requestToken",
+            params, httpApi.PREFIX_IDENTITY_V2, identityAccessToken,
+        );
+        // TODO: Fold callback into above call once v1 path below is removed
+        if (callback) callback(null, response);
+        return response;
+    } catch (err) {
+        if (err.cors === "rejected" || err.httpStatus === 404) {
+            // Fall back to deprecated v1 API for now
+            // TODO: Remove this path once v2 is only supported version
+            // See https://github.com/vector-im/riot-web/issues/10443
+            logger.warn("IS doesn't support v2, falling back to deprecated v1");
+            return await this._http.idServerRequest(
+                callback, "POST", "/validate/email/requestToken",
+                params, httpApi.PREFIX_IDENTITY_V1,
+            );
+        }
+        if (callback) callback(err);
+        throw err;
+    }
 };

 /**
- * Submits an MSISDN token to the identity server
+ * Requests a MSISDN verification token directly from an identity server.
+ *
+ * This API is used as part of binding a MSISDN for discovery on an identity
+ * server. The validation data that results should be passed to the
+ * `bindThreePid` method to complete the binding process.
+ *
+ * @param {string} phoneCountry The ISO 3166-1 alpha-2 code for the country in
+ *                 which phoneNumber should be parsed relative to.
+ * @param {string} phoneNumber The phone number, in national or international
+ *                 format
+ * @param {string} clientSecret A secret binary string generated by the client.
+ *                 It is recommended this be around 16 ASCII characters.
+ * @param {number} sendAttempt If an identity server sees a duplicate request
+ *                 with the same sendAttempt, it will not send another SMS.
+ *                 To request another SMS to be sent, use a larger value for
+ *                 the sendAttempt param as was used in the previous request.
+ * @param {string} nextLink Optional If specified, the client will be redirected
+ *                 to this link after validation.
+ * @param {module:client.callback} callback Optional.
+ * @param {string} identityAccessToken The `access_token` field of the Identity
+ * Server `/account/register` response (see {@link registerWithIdentityServer}).
+ *
+ * @return {module:client.Promise} Resolves: TODO
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ * @throws Error if no identity server is set
+ */
+MatrixBaseApis.prototype.requestMsisdnToken = async function(
+    phoneCountry,
+    phoneNumber,
+    clientSecret,
+    sendAttempt,
+    nextLink,
+    callback,
+    identityAccessToken,
+) {
+    const params = {
+        client_secret: clientSecret,
+        country: phoneCountry,
+        phone_number: phoneNumber,
+        send_attempt: sendAttempt,
+        next_link: nextLink,
+    };
+
+    try {
+        const response = await this._http.idServerRequest(
+            undefined, "POST", "/validate/msisdn/requestToken",
+            params, httpApi.PREFIX_IDENTITY_V2, identityAccessToken,
+        );
+        // TODO: Fold callback into above call once v1 path below is removed
+        if (callback) callback(null, response);
+        return response;
+    } catch (err) {
+        if (err.cors === "rejected" || err.httpStatus === 404) {
+            // Fall back to deprecated v1 API for now
+            // TODO: Remove this path once v2 is only supported version
+            // See https://github.com/vector-im/riot-web/issues/10443
+            logger.warn("IS doesn't support v2, falling back to deprecated v1");
+            return await this._http.idServerRequest(
+                callback, "POST", "/validate/msisdn/requestToken",
+                params, httpApi.PREFIX_IDENTITY_V1,
+            );
+        }
+        if (callback) callback(err);
+        throw err;
+    }
+};
+
+/**
+ * Submits a MSISDN token to the identity server
 *
 * This is used when submitting the code sent by SMS to a phone number.
 * The ID server has an equivalent API for email but the js-sdk does
@@ -1732,46 +1973,323 @@ MatrixBaseApis.prototype.requestEmailToken = function(email, clientSecret,
 * @param {string} sid The sid given in the response to requestToken
 * @param {string} clientSecret A secret binary string generated by the client.
 *                 This must be the same value submitted in the requestToken call.
- * @param {string} token The token, as enetered by the user.
+ * @param {string} msisdnToken The MSISDN token, as enetered by the user.
+ * @param {string} identityAccessToken The `access_token` field of the Identity
+ * Server `/account/register` response (see {@link registerWithIdentityServer}).
 *
 * @return {module:client.Promise} Resolves: Object, currently with no parameters.
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 * @throws Error if No ID server is set
 */
-MatrixBaseApis.prototype.submitMsisdnToken = function(sid, clientSecret, token) {
+MatrixBaseApis.prototype.submitMsisdnToken = async function(
+    sid,
+    clientSecret,
+    msisdnToken,
+    identityAccessToken,
+) {
    const params = {
        sid: sid,
        client_secret: clientSecret,
-        token: token,
+        token: msisdnToken,
    };
-    return this._http.idServerRequest(
-        undefined, "POST", "/validate/msisdn/submitToken",
-        params, httpApi.PREFIX_IDENTITY_V1,
+
+    try {
+        return await this._http.idServerRequest(
+            undefined, "POST", "/validate/msisdn/submitToken",
+            params, httpApi.PREFIX_IDENTITY_V2, identityAccessToken,
+        );
+    } catch (err) {
+        if (err.cors === "rejected" || err.httpStatus === 404) {
+            // Fall back to deprecated v1 API for now
+            // TODO: Remove this path once v2 is only supported version
+            // See https://github.com/vector-im/riot-web/issues/10443
+            logger.warn("IS doesn't support v2, falling back to deprecated v1");
+            return await this._http.idServerRequest(
+                undefined, "POST", "/validate/msisdn/submitToken",
+                params, httpApi.PREFIX_IDENTITY_V1,
+            );
+        }
+        throw err;
+    }
+};
+
+/**
+ * Submits a MSISDN token to an arbitrary URL.
+ *
+ * This is used when submitting the code sent by SMS to a phone number in the
+ * newer 3PID flow where the homeserver validates 3PID ownership (as part of
+ * `requestAdd3pidMsisdnToken`). The homeserver response may include a
+ * `submit_url` to specify where the token should be sent, and this helper can
+ * be used to pass the token to this URL.
+ *
+ * @param {string} url The URL to submit the token to
+ * @param {string} sid The sid given in the response to requestToken
+ * @param {string} clientSecret A secret binary string generated by the client.
+ *                 This must be the same value submitted in the requestToken call.
+ * @param {string} msisdnToken The MSISDN token, as enetered by the user.
+ *
+ * @return {module:client.Promise} Resolves: Object, currently with no parameters.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.submitMsisdnTokenOtherUrl = function(
+    url,
+    sid,
+    clientSecret,
+    msisdnToken,
+) {
+    const params = {
+        sid: sid,
+        client_secret: clientSecret,
+        token: msisdnToken,
+    };
+
+    return this._http.requestOtherUrl(
+        undefined, "POST", url, undefined, params,
    );
 };

+/**
+ * Gets the V2 hashing information from the identity server. Primarily useful for
+ * lookups.
+ * @param {string} identityAccessToken The access token for the identity server.
+ * @returns {Promise<object>} The hashing information for the identity server.
+ */
+MatrixBaseApis.prototype.getIdentityHashDetails = function(identityAccessToken) {
+    return this._http.idServerRequest(
+        undefined, "GET", "/hash_details",
+        null, httpApi.PREFIX_IDENTITY_V2, identityAccessToken,
+    );
+};
+
+/**
+ * Performs a hashed lookup of addresses against the identity server. This is
+ * only supported on identity servers which have at least the version 2 API.
+ * @param {Array<Array<string,string>>} addressPairs An array of 2 element arrays.
+ * The first element of each pair is the address, the second is the 3PID medium.
+ * Eg: ["email@example.org", "email"]
+ * @param {string} identityAccessToken The access token for the identity server.
+ * @returns {Promise<Array<{address, mxid}>>} A collection of address mappings to
+ * found MXIDs. Results where no user could be found will not be listed.
+ */
+MatrixBaseApis.prototype.identityHashedLookup = async function(
+    addressPairs, // [["email@example.org", "email"], ["10005550000", "msisdn"]]
+    identityAccessToken,
+) {
+    const params = {
+        // addresses: ["email@example.org", "10005550000"],
+        // algorithm: "sha256",
+        // pepper: "abc123"
+    };
+
+    // Get hash information first before trying to do a lookup
+    const hashes = await this.getIdentityHashDetails(identityAccessToken);
+    if (!hashes || !hashes['lookup_pepper'] || !hashes['algorithms']) {
+        throw new Error("Unsupported identity server: bad response");
+    }
+
+    params['pepper'] = hashes['lookup_pepper'];
+
+    const localMapping = {
+        // hashed identifier => plain text address
+        // For use in this function's return format
+    };
+
+    // When picking an algorithm, we pick the hashed over no hashes
+    if (hashes['algorithms'].includes('sha256')) {
+        // Abuse the olm hashing
+        const olmutil = new global.Olm.Utility();
+        params["addresses"] = addressPairs.map(p => {
+            const addr = p[0].toLowerCase(); // lowercase to get consistent hashes
+            const med = p[1].toLowerCase();
+            const hashed = olmutil.sha256(`${addr} ${med} ${params['pepper']}`)
+                .replace(/\+/g, '-').replace(/\//g, '_'); // URL-safe base64
+            // Map the hash to a known (case-sensitive) address. We use the case
+            // sensitive version because the caller might be expecting that.
+            localMapping[hashed] = p[0];
+            return hashed;
+        });
+        params["algorithm"] = "sha256";
+    } else if (hashes['algorithms'].includes('none')) {
+        params["addresses"] = addressPairs.map(p => {
+            const addr = p[0].toLowerCase(); // lowercase to get consistent hashes
+            const med = p[1].toLowerCase();
+            const unhashed = `${addr} ${med}`;
+            // Map the unhashed values to a known (case-sensitive) address. We use
+            // the case sensitive version because the caller might be expecting that.
+            localMapping[unhashed] = p[0];
+            return unhashed;
+        });
+        params["algorithm"] = "none";
+    } else {
+        throw new Error("Unsupported identity server: unknown hash algorithm");
+    }
+
+    const response = await this._http.idServerRequest(
+        undefined, "POST", "/lookup",
+        params, httpApi.PREFIX_IDENTITY_V2, identityAccessToken,
+    );
+
+    if (!response || !response['mappings']) return []; // no results
+
+    const foundAddresses = [/* {address: "plain@example.org", mxid} */];
+    for (const hashed of Object.keys(response['mappings'])) {
+        const mxid = response['mappings'][hashed];
+        const plainAddress = localMapping[hashed];
+        if (!plainAddress) {
+            throw new Error("Identity server returned more results than expected");
+        }
+
+        foundAddresses.push({address: plainAddress, mxid});
+    }
+    return foundAddresses;
+};
+
 /**
 * Looks up the public Matrix ID mapping for a given 3rd party
 * identifier from the Identity Server
+ *
 * @param {string} medium The medium of the threepid, eg. 'email'
 * @param {string} address The textual address of the threepid
 * @param {module:client.callback} callback Optional.
+ * @param {string} identityAccessToken The `access_token` field of the Identity
+ * Server `/account/register` response (see {@link registerWithIdentityServer}).
+ *
 * @return {module:client.Promise} Resolves: A threepid mapping
 *                                 object or the empty object if no mapping
 *                                 exists
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
-MatrixBaseApis.prototype.lookupThreePid = function(medium, address, callback) {
-    const params = {
-        medium: medium,
-        address: address,
-    };
-    return this._http.idServerRequest(
-        callback, "GET", "/lookup",
-        params, httpApi.PREFIX_IDENTITY_V1,
-    );
+MatrixBaseApis.prototype.lookupThreePid = async function(
+    medium,
+    address,
+    callback,
+    identityAccessToken,
+) {
+    try {
+        // Note: we're using the V2 API by calling this function, but our
+        // function contract requires a V1 response. We therefore have to
+        // convert it manually.
+        const response = await this.identityHashedLookup(
+            [[address, medium]], identityAccessToken,
+        );
+        const result = response.find(p => p.address === address);
+        if (!result) {
+            // TODO: Fold callback into above call once v1 path below is removed
+            if (callback) callback(null, {});
+            return {};
+        }
+
+        const mapping = {
+            address,
+            medium,
+            mxid: result.mxid,
+
+            // We can't reasonably fill these parameters:
+            // not_before
+            // not_after
+            // ts
+            // signatures
+        };
+
+        // TODO: Fold callback into above call once v1 path below is removed
+        if (callback) callback(null, mapping);
+        return mapping;
+    } catch (err) {
+        if (err.cors === "rejected" || err.httpStatus === 404) {
+            // Fall back to deprecated v1 API for now
+            // TODO: Remove this path once v2 is only supported version
+            // See https://github.com/vector-im/riot-web/issues/10443
+            const params = {
+                medium: medium,
+                address: address,
+            };
+            logger.warn("IS doesn't support v2, falling back to deprecated v1");
+            return await this._http.idServerRequest(
+                callback, "GET", "/lookup",
+                params, httpApi.PREFIX_IDENTITY_V1,
+            );
+        }
+        if (callback) callback(err, undefined);
+        throw err;
+    }
 };

+/**
+ * Looks up the public Matrix ID mappings for multiple 3PIDs.
+ *
+ * @param {Array.<Array.<string>>} query Array of arrays containing
+ * [medium, address]
+ * @param {string} identityAccessToken The `access_token` field of the Identity
+ * Server `/account/register` response (see {@link registerWithIdentityServer}).
+ *
+ * @return {module:client.Promise} Resolves: Lookup results from IS.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.bulkLookupThreePids = async function(
+    query,
+    identityAccessToken,
+) {
+    try {
+        // Note: we're using the V2 API by calling this function, but our
+        // function contract requires a V1 response. We therefore have to
+        // convert it manually.
+        const response = await this.identityHashedLookup(
+            // We have to reverse the query order to get [address, medium] pairs
+            query.map(p => [p[1], p[0]]), identityAccessToken,
+        );
+
+        const v1results = [];
+        for (const mapping of response) {
+            const originalQuery = query.find(p => p[1] === mapping.address);
+            if (!originalQuery) {
+                throw new Error("Identity sever returned unexpected results");
+            }
+
+            v1results.push([
+                originalQuery[0], // medium
+                mapping.address,
+                mapping.mxid,
+            ]);
+        }
+
+        return {threepids: v1results};
+    } catch (err) {
+        if (err.cors === "rejected" || err.httpStatus === 404) {
+            // Fall back to deprecated v1 API for now
+            // TODO: Remove this path once v2 is only supported version
+            // See https://github.com/vector-im/riot-web/issues/10443
+            const params = {
+                threepids: query,
+            };
+            logger.warn("IS doesn't support v2, falling back to deprecated v1");
+            return await this._http.idServerRequest(
+                undefined, "POST", "/bulk_lookup", params,
+                httpApi.PREFIX_IDENTITY_V1, identityAccessToken,
+            );
+        }
+        throw err;
+    }
+};
+
+/**
+ * Get account info from the Identity Server. This is useful as a neutral check
+ * to verify that other APIs are likely to approve access by testing that the
+ * token is valid, terms have been agreed, etc.
+ *
+ * @param {string} identityAccessToken The `access_token` field of the Identity
+ * Server `/account/register` response (see {@link registerWithIdentityServer}).
+ *
+ * @return {module:client.Promise} Resolves: an object with account info.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.getIdentityAccount = function(
+    identityAccessToken,
+) {
+    return this._http.idServerRequest(
+        undefined, "GET", "/account",
+        undefined, httpApi.PREFIX_IDENTITY_V2, identityAccessToken,
+    );
+};

 // Direct-to-device messaging
 // ==========================
@@ -1855,6 +2373,42 @@ MatrixBaseApis.prototype.getThirdpartyUser = function(protocol, params) {
    return this._http.authedRequest(undefined, "GET", path, params, undefined);
 };

+MatrixBaseApis.prototype.getTerms = function(serviceType, baseUrl) {
+    const url = termsUrlForService(serviceType, baseUrl);
+    return this._http.requestOtherUrl(
+        undefined, 'GET', url,
+    );
+};
+
+MatrixBaseApis.prototype.agreeToTerms = function(
+    serviceType, baseUrl, accessToken, termsUrls,
+) {
+    const url = termsUrlForService(serviceType, baseUrl);
+    const headers = {
+        Authorization: "Bearer " + accessToken,
+    };
+    return this._http.requestOtherUrl(
+        undefined, 'POST', url, null, { user_accepts: termsUrls }, { headers },
+    );
+};
+
+/**
+ * Reports an event as inappropriate to the server, which may then notify the appropriate people.
+ * @param {string} roomId The room in which the event being reported is located.
+ * @param {string} eventId The event to report.
+ * @param {number} score The score to rate this content as where -100 is most offensive and 0 is inoffensive.
+ * @param {string} reason The reason the content is being reported. May be blank.
+ * @returns {module:client.Promise} Resolves to an empty object if successful
+ */
+MatrixBaseApis.prototype.reportEvent = function(roomId, eventId, score, reason) {
+    const path = utils.encodeUri("/rooms/$roomId/report/$eventId", {
+        $roomId: roomId,
+        $eventId: eventId,
+    });
+
+    return this._http.authedRequest(undefined, "POST", path, null, {score, reason});
+};
+
 /**
 * MatrixBaseApis object
 */
@@ -46,7 +46,7 @@ const olmlib = require("./crypto/olmlib");

 import ReEmitter from './ReEmitter';
 import RoomList from './crypto/RoomList';
-import logger from '../src/logger';
+import logger from './logger';

 import Crypto from './crypto';
 import { isCryptoAvailable } from './crypto';
@@ -108,6 +108,15 @@ function keyFromRecoverySession(session, decryptionKey) {
 *
 * @param {string} opts.userId The user ID for this user.
 *
+ * @param {IdentityServerProvider} [opts.identityServer]
+ * Optional. A provider object with one function `getAccessToken`, which is a
+ * callback that returns a Promise<String> of an identity access token to supply
+ * with identity requests. If the object is unset, no access token will be
+ * supplied.
+ * See also https://github.com/vector-im/riot-web/issues/10615 which seeks to
+ * replace the previous approach of manual access tokens params with this
+ * callback throughout the SDK.
+ *
 * @param {Object=} opts.store
 *    The data store used for sync data from the homeserver. If not specified,
 *    this client will not store any HTTP responses. The `createClient` helper
@@ -159,17 +168,17 @@ function keyFromRecoverySession(session, decryptionKey) {
 * that the application can handle.  Each element should be an item from {@link
 * module:crypto~verificationMethods verificationMethods}, or a class that
 * implements the {$link module:crypto/verification/Base verifier interface}.
+ *
+ * @param {boolean} [opts.forceTURN]
+ * Optional. Whether relaying calls through a TURN server should be forced.
+ *
+ * @param {boolean} [opts.fallbackICEServerAllowed]
+ * Optional. Whether to allow a fallback ICE server should be used for negotiating a
+ * WebRTC connection if the homeserver doesn't provide any servers. Defaults to false.
 */
 function MatrixClient(opts) {
-    // Allow trailing slash in HS url
-    if (opts.baseUrl && opts.baseUrl.endsWith("/")) {
-        opts.baseUrl = opts.baseUrl.substr(0, opts.baseUrl.length - 1);
-    }
-
-    // Allow trailing slash in IS url
-    if (opts.idBaseUrl && opts.idBaseUrl.endsWith("/")) {
-        opts.idBaseUrl = opts.idBaseUrl.substr(0, opts.idBaseUrl.length - 1);
-    }
+    opts.baseUrl = utils.ensureNoTrailingSlash(opts.baseUrl);
+    opts.idBaseUrl = utils.ensureNoTrailingSlash(opts.idBaseUrl);

    MatrixBaseApis.call(this, opts);

@@ -228,6 +237,7 @@ function MatrixClient(opts) {
    this._verificationMethods = opts.verificationMethods;

    this._forceTURN = opts.forceTURN || false;
+    this._fallbackICEServerAllowed = opts.fallbackICEServerAllowed || false;

    // List of which rooms have encryption enabled: separate from crypto because
    // we still want to know which rooms are encrypted even if crypto is disabled:
@@ -237,7 +247,9 @@ function MatrixClient(opts) {
    // The pushprocessor caches useful things, so keep one and re-use it
    this._pushProcessor = new PushProcessor(this);

-    this._serverSupportsLazyLoading = null;
+    // Cache of the server's /versions response
+    // TODO: This should expire: https://github.com/matrix-org/matrix-js-sdk/issues/1020
+    this._serverVersionsCache = null;

    this._cachedCapabilities = null; // { capabilities: {}, lastUpdated: timestamp }

@@ -773,6 +785,40 @@ async function _setDeviceVerification(
    client.emit("deviceVerificationChanged", userId, deviceId, dev);
 }

+/**
+ * Request a key verification from another user, using a DM.
+ *
+ * @param {string} userId the user to request verification with
+ * @param {string} roomId the room to use for verification
+ * @param {Array} methods array of verification methods to use.  Defaults to
+ *    all known methods
+ *
+ * @returns {Promise<module:crypto/verification/Base>} resolves to a verifier
+ *    when the request is accepted by the other user
+ */
+MatrixClient.prototype.requestVerificationDM = function(userId, roomId, methods) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+    return this._crypto.requestVerificationDM(userId, roomId, methods);
+};
+
+/**
+ * Accept a key verification request from a DM.
+ *
+ * @param {module:models/event~MatrixEvent} event the verification request
+ * that is accepted
+ * @param {string} method the verification mmethod to use
+ *
+ * @returns {module:crypto/verification/Base} a verifier
+ */
+MatrixClient.prototype.acceptVerificationDM = function(event, method) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+    return this._crypto.acceptVerificationDM(event, method);
+};
+
 /**
 * Request a key verification from another user.
 *
@@ -1369,8 +1415,10 @@ MatrixClient.prototype.getGroups = function() {
 * @return {module:client.Promise} Resolves with an object containing the config.
 */
 MatrixClient.prototype.getMediaConfig = function(callback) {
-    return this._http.authedRequestWithPrefix(
-        callback, "GET", "/config", undefined, undefined, httpApi.PREFIX_MEDIA_R0,
+    return this._http.authedRequest(
+        callback, "GET", "/config", undefined, undefined, {
+            prefix: httpApi.PREFIX_MEDIA_R0,
+        },
    );
 };

@@ -2178,11 +2226,17 @@ MatrixClient.prototype.sendHtmlEmote = function(roomId, body, htmlBody, callback
 * Send a receipt.
 * @param {Event} event The event being acknowledged
 * @param {string} receiptType The kind of receipt e.g. "m.read"
+ * @param {object} opts Additional content to send alongside the receipt.
 * @param {module:client.callback} callback Optional.
 * @return {module:client.Promise} Resolves: TODO
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
-MatrixClient.prototype.sendReceipt = function(event, receiptType, callback) {
+MatrixClient.prototype.sendReceipt = function(event, receiptType, opts, callback) {
+    if (typeof(opts) === 'function') {
+        callback = opts;
+        opts = {};
+    }
+
    if (this.isGuest()) {
        return Promise.resolve({}); // guests cannot send receipts so don't bother.
    }
@@ -2193,7 +2247,7 @@ MatrixClient.prototype.sendReceipt = function(event, receiptType, callback) {
        $eventId: event.getId(),
    });
    const promise = this._http.authedRequest(
-        callback, "POST", path, undefined, {},
+        callback, "POST", path, undefined, opts || {},
    );

    const room = this.getRoom(event.getRoomId());
@@ -2206,17 +2260,32 @@ MatrixClient.prototype.sendReceipt = function(event, receiptType, callback) {
 /**
 * Send a read receipt.
 * @param {Event} event The event that has been read.
+ * @param {object} opts The options for the read receipt.
+ * @param {boolean} opts.hidden True to prevent the receipt from being sent to
+ * other users and homeservers. Default false (send to everyone). <b>This
+ * property is unstable and may change in the future.</b>
 * @param {module:client.callback} callback Optional.
 * @return {module:client.Promise} Resolves: TODO
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
-MatrixClient.prototype.sendReadReceipt = async function(event, callback) {
+MatrixClient.prototype.sendReadReceipt = async function(event, opts, callback) {
+    if (typeof(opts) === 'function') {
+        callback = opts;
+        opts = {};
+    }
+    if (!opts) opts = {};
+
    const eventId = event.getId();
    const room = this.getRoom(event.getRoomId());
    if (room && room.hasPendingEvent(eventId)) {
        throw new Error(`Cannot set read receipt to a pending event (${eventId})`);
    }
-    return this.sendReceipt(event, "m.read", callback);
+
+    const addlContent = {
+        "m.hidden": Boolean(opts.hidden),
+    };
+
+    return this.sendReceipt(event, "m.read", addlContent, callback);
 };

 /**
@@ -2229,9 +2298,14 @@ MatrixClient.prototype.sendReadReceipt = async function(event, callback) {
 * @param {string} rrEvent the event tracked by the read receipt. This is here for
 * convenience because the RR and the RM are commonly updated at the same time as each
 * other. The local echo of this receipt will be done if set. Optional.
+ * @param {object} opts Options for the read markers
+ * @param {object} opts.hidden True to hide the receipt from other users and homeservers.
+ * <b>This property is unstable and may change in the future.</b>
 * @return {module:client.Promise} Resolves: the empty object, {}.
 */
-MatrixClient.prototype.setRoomReadMarkers = async function(roomId, rmEventId, rrEvent) {
+MatrixClient.prototype.setRoomReadMarkers = async function(
+    roomId, rmEventId, rrEvent, opts,
+) {
    const room = this.getRoom(roomId);
    if (room && room.hasPendingEvent(rmEventId)) {
        throw new Error(`Cannot set read marker to a pending event (${rmEventId})`);
@@ -2249,7 +2323,7 @@ MatrixClient.prototype.setRoomReadMarkers = async function(roomId, rmEventId, rr
        }
    }

-    return this.setRoomReadMarkersHttpRequest(roomId, rmEventId, rrEventId);
+    return this.setRoomReadMarkersHttpRequest(roomId, rmEventId, rrEventId, opts);
 };

 /**
@@ -2275,11 +2349,13 @@ MatrixClient.prototype.getUrlPreview = function(url, ts, callback) {
    }

    const self = this;
-    return this._http.authedRequestWithPrefix(
+    return this._http.authedRequest(
        callback, "GET", "/preview_url", {
            url: url,
            ts: ts,
-        }, undefined, httpApi.PREFIX_MEDIA_R0,
+        }, undefined, {
+            prefix: httpApi.PREFIX_MEDIA_R0,
+        },
    ).then(function(response) {
        // TODO: expire cache occasionally
        self.urlPreviewCache[key] = response;
@@ -2369,6 +2445,7 @@ MatrixClient.prototype.getRoomUpgradeHistory = function(roomId, verifyLinks=fals
    while (tombstoneEvent) {
        const refRoom = this.getRoom(tombstoneEvent.getContent()['replacement_room']);
        if (!refRoom) break; // end of the chain
+        if (refRoom.roomId === currentRoom.roomId) break; // Tombstone is referencing it's own room

        if (verifyLinks) {
            createEvent = refRoom.currentState.getStateEvents("m.room.create", "");
@@ -2380,6 +2457,12 @@ MatrixClient.prototype.getRoomUpgradeHistory = function(roomId, verifyLinks=fals

        // Push to the end because we're looking forwards
        upgradeHistory.push(refRoom);
+        const roomIds = new Set(upgradeHistory.map((ref) => ref.roomId));
+        if (roomIds.size < upgradeHistory.length) {
+           // The last room added to the list introduced a previous roomId
+           // To avoid recursion, return the last rooms - 1
+           return upgradeHistory.slice(0, upgradeHistory.length - 1);
+        }

        // Set the current room to the reference room so we know where we're at
        currentRoom = refRoom;
@@ -2424,7 +2507,12 @@ MatrixClient.prototype.inviteByEmail = function(roomId, email, callback) {
 * @return {module:client.Promise} Resolves: TODO
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
-MatrixClient.prototype.inviteByThreePid = function(roomId, medium, address, callback) {
+MatrixClient.prototype.inviteByThreePid = async function(
+    roomId,
+    medium,
+    address,
+    callback,
+) {
    const path = utils.encodeUri(
        "/rooms/$roomId/invite",
        { $roomId: roomId },
@@ -2437,12 +2525,24 @@ MatrixClient.prototype.inviteByThreePid = function(roomId, medium, address, call
            errcode: "ORG.MATRIX.JSSDK_MISSING_PARAM",
        }));
    }
-
-    return this._http.authedRequest(callback, "POST", path, undefined, {
+    const params = {
        id_server: identityServerUrl,
        medium: medium,
        address: address,
-    });
+    };
+
+    if (
+        this.identityServer &&
+        this.identityServer.getAccessToken &&
+        await this.doesServerAcceptIdentityAccessToken()
+    ) {
+        const identityAccessToken = await this.identityServer.getAccessToken();
+        if (identityAccessToken) {
+            params.id_access_token = identityAccessToken;
+        }
+    }
+
+    return this._http.authedRequest(callback, "POST", path, undefined, params);
 };

 /**
@@ -3231,14 +3331,9 @@ MatrixClient.prototype.setGuestAccess = function(roomId, opts) {

 /**
 * Requests an email verification token for the purposes of registration.
- * This API proxies the Identity Server /validate/email/requestToken API,
- * adding registration-specific behaviour. Specifically, if an account with
- * the given email address already exists, it will either send an email
- * to the address informing them of this or return M_THREEPID_IN_USE
- * (which one is up to the Home Server).
- *
- * requestEmailToken calls the equivalent API directly on the ID server,
- * therefore bypassing the registration-specific logic.
+ * This API requests a token from the homeserver.
+ * The doesServerRequireIdServerParam() method can be used to determine if
+ * the server requires the id_server parameter to be provided.
 *
 * Parameters and return value are as for requestEmailToken

@@ -3263,8 +3358,9 @@ MatrixClient.prototype.requestRegisterEmailToken = function(email, clientSecret,

 /**
 * Requests a text message verification token for the purposes of registration.
- * This API proxies the Identity Server /validate/msisdn/requestToken API,
- * adding registration-specific behaviour, as with requestRegisterEmailToken.
+ * This API requests a token from the homeserver.
+ * The doesServerRequireIdServerParam() method can be used to determine if
+ * the server requires the id_server parameter to be provided.
 *
 * @param {string} phoneCountry The ISO 3166-1 alpha-2 code for the country in which
 *    phoneNumber should be parsed relative to.
@@ -3291,15 +3387,13 @@ MatrixClient.prototype.requestRegisterMsisdnToken = function(phoneCountry, phone
 /**
 * Requests an email verification token for the purposes of adding a
 * third party identifier to an account.
- * This API proxies the Identity Server /validate/email/requestToken API,
- * adding specific behaviour for the addition of email addresses to an
- * account. Specifically, if an account with
- * the given email address already exists, it will either send an email
- * to the address informing them of this or return M_THREEPID_IN_USE
- * (which one is up to the Home Server).
- *
- * requestEmailToken calls the equivalent API directly on the ID server,
- * therefore bypassing the email addition specific logic.
+ * This API requests a token from the homeserver.
+ * The doesServerRequireIdServerParam() method can be used to determine if
+ * the server requires the id_server parameter to be provided.
+ * If an account with the given email address already exists and is
+ * associated with an account other than the one the user is authed as,
+ * it will either send an email to the address informing them of this
+ * or return M_THREEPID_IN_USE (which one is up to the Home Server).
 *
 * @param {string} email As requestEmailToken
 * @param {string} clientSecret As requestEmailToken
@@ -3415,15 +3509,30 @@ MatrixClient.prototype.requestPasswordMsisdnToken = function(phoneCountry, phone
 * @param {object} params Parameters for the POST request
 * @return {module:client.Promise} Resolves: As requestEmailToken
 */
-MatrixClient.prototype._requestTokenFromEndpoint = function(endpoint, params) {
-    const id_server_url = url.parse(this.idBaseUrl);
-    if (id_server_url.host === null) {
-        throw new Error("Invalid ID server URL: " + this.idBaseUrl);
+MatrixClient.prototype._requestTokenFromEndpoint = async function(endpoint, params) {
+    const postParams = Object.assign({}, params);
+
+    // If the HS supports separate add and bind, then requestToken endpoints
+    // don't need an IS as they are all validated by the HS directly.
+    if (!await this.doesServerSupportSeparateAddAndBind() && this.idBaseUrl) {
+        const idServerUrl = url.parse(this.idBaseUrl);
+        if (!idServerUrl.host) {
+            throw new Error("Invalid ID server URL: " + this.idBaseUrl);
+        }
+        postParams.id_server = idServerUrl.host;
+
+        if (
+            this.identityServer &&
+            this.identityServer.getAccessToken &&
+            await this.doesServerAcceptIdentityAccessToken()
+        ) {
+            const identityAccessToken = await this.identityServer.getAccessToken();
+            if (identityAccessToken) {
+                postParams.id_access_token = identityAccessToken;
+            }
+        }
    }

-    const postParams = Object.assign({}, params, {
-        id_server: id_server_url.host,
-    });
    return this._http.request(
        undefined, "POST", endpoint, undefined,
        postParams,
@@ -3879,6 +3988,77 @@ MatrixClient.prototype.getTurnServers = function() {
    return this._turnServers || [];
 };

+/**
+ * Set whether to allow a fallback ICE server should be used for negotiating a
+ * WebRTC connection if the homeserver doesn't provide any servers. Defaults to
+ * false.
+ *
+ * @param {boolean} allow
+ */
+MatrixClient.prototype.setFallbackICEServerAllowed = function(allow) {
+    this._fallbackICEServerAllowed = allow;
+};
+
+/**
+ * Get whether to allow a fallback ICE server should be used for negotiating a
+ * WebRTC connection if the homeserver doesn't provide any servers. Defaults to
+ * false.
+ *
+ * @returns {boolean}
+ */
+MatrixClient.prototype.isFallbackICEServerAllowed = function() {
+    return this._fallbackICEServerAllowed;
+};
+
+// Synapse-specific APIs
+// =====================
+
+/**
+ * Determines if the current user is an administrator of the Synapse homeserver.
+ * Returns false if untrue or the homeserver does not appear to be a Synapse
+ * homeserver. <strong>This function is implementation specific and may change
+ * as a result.</strong>
+ * @return {boolean} true if the user appears to be a Synapse administrator.
+ */
+MatrixClient.prototype.isSynapseAdministrator = function() {
+    return this.whoisSynapseUser(this.getUserId())
+        .then(() => true)
+        .catch(() => false);
+};
+
+/**
+ * Performs a whois lookup on a user using Synapse's administrator API.
+ * <strong>This function is implementation specific and may change as a
+ * result.</strong>
+ * @param {string} userId the User ID to look up.
+ * @return {object} the whois response - see Synapse docs for information.
+ */
+MatrixClient.prototype.whoisSynapseUser = function(userId) {
+    const path = utils.encodeUri(
+        "/_synapse/admin/v1/whois/$userId",
+        { $userId: userId },
+    );
+    return this._http.authedRequest(
+        undefined, 'GET', path, undefined, undefined, {prefix: ''},
+    );
+};
+
+/**
+ * Deactivates a user using Synapse's administrator API. <strong>This
+ * function is implementation specific and may change as a result.</strong>
+ * @param {string} userId the User ID to deactivate.
+ * @return {object} the deactivate response - see Synapse docs for information.
+ */
+MatrixClient.prototype.deactivateSynapseUser = function(userId) {
+    const path = utils.encodeUri(
+        "/_synapse/admin/v1/deactivate/$userId",
+        { $userId: userId },
+    );
+    return this._http.authedRequest(
+        undefined, 'POST', path, undefined, undefined, {prefix: ''},
+    );
+};
+
 // Higher level APIs
 // =================

@@ -4010,13 +4190,14 @@ MatrixClient.prototype.stopClient = function() {
    global.clearTimeout(this._checkTurnServersTimeoutID);
 };

-/*
- * Query the server to see if it support members lazy loading
- * @return {Promise<boolean>} true if server supports lazy loading
+/**
+ * Get the API versions supported by the server, along with any
+ * unstable APIs it supports
+ * @return {Promise<object>} The server /versions response
 */
-MatrixClient.prototype.doesServerSupportLazyLoading = async function() {
-    if (this._serverSupportsLazyLoading === null) {
-        const response = await this._http.request(
+MatrixClient.prototype.getVersions = async function() {
+    if (this._serverVersionsCache === null) {
+        this._serverVersionsCache = await this._http.request(
            undefined, // callback
            "GET", "/_matrix/client/versions",
            undefined, // queryParams
@@ -4025,18 +4206,80 @@ MatrixClient.prototype.doesServerSupportLazyLoading = async function() {
                prefix: '',
            },
        );
-
-        const versions = response["versions"];
-        const unstableFeatures = response["unstable_features"];
-
-        this._serverSupportsLazyLoading =
-            (versions && versions.includes("r0.5.0"))
-            || (unstableFeatures && unstableFeatures["m.lazy_load_members"]);
    }
-    return this._serverSupportsLazyLoading;
+    return this._serverVersionsCache;
 };

-/*
+/**
+ * Query the server to see if it support members lazy loading
+ * @return {Promise<boolean>} true if server supports lazy loading
+ */
+MatrixClient.prototype.doesServerSupportLazyLoading = async function() {
+    const response = await this.getVersions();
+
+    const versions = response["versions"];
+    const unstableFeatures = response["unstable_features"];
+
+    return (versions && versions.includes("r0.5.0"))
+        || (unstableFeatures && unstableFeatures["m.lazy_load_members"]);
+};
+
+/**
+ * Query the server to see if the `id_server` parameter is required
+ * when registering with an 3pid, adding a 3pid or resetting password.
+ * @return {Promise<boolean>} true if id_server parameter is required
+ */
+MatrixClient.prototype.doesServerRequireIdServerParam = async function() {
+    const response = await this.getVersions();
+
+    const versions = response["versions"];
+
+    // Supporting r0.6.0 is the same as having the flag set to false
+    if (versions && versions.includes("r0.6.0")) {
+        return false;
+    }
+
+    const unstableFeatures = response["unstable_features"];
+    if (unstableFeatures["m.require_identity_server"] === undefined) {
+        return true;
+    } else {
+        return unstableFeatures["m.require_identity_server"];
+    }
+};
+
+/**
+ * Query the server to see if the `id_access_token` parameter can be safely
+ * passed to the homeserver. Some homeservers may trigger errors if they are not
+ * prepared for the new parameter.
+ * @return {Promise<boolean>} true if id_access_token can be sent
+ */
+MatrixClient.prototype.doesServerAcceptIdentityAccessToken = async function() {
+    const response = await this.getVersions();
+
+    const versions = response["versions"];
+    const unstableFeatures = response["unstable_features"];
+
+    return (versions && versions.includes("r0.6.0"))
+        || (unstableFeatures && unstableFeatures["m.id_access_token"]);
+};
+
+/**
+ * Query the server to see if it supports separate 3PID add and bind functions.
+ * This affects the sequence of API calls clients should use for these operations,
+ * so it's helpful to be able to check for support.
+ * @return {Promise<boolean>} true if separate functions are supported
+ */
+MatrixClient.prototype.doesServerSupportSeparateAddAndBind = async function() {
+    const response = await this.getVersions();
+
+    const versions = response["versions"];
+    const unstableFeatures = response["unstable_features"];
+
+    return (versions && versions.includes("r0.6.0"))
+        || (unstableFeatures && unstableFeatures["m.separate_add_and_bind"]);
+};
+
+/**
 * Get if lazy loading members is being used.
 * @return {boolean} Whether or not members are lazy loaded by this client
 */
@@ -4044,7 +4287,7 @@ MatrixClient.prototype.hasLazyLoadMembersEnabled = function() {
    return !!this._clientOpts.lazyLoadMembers;
 };

-/*
+/**
 * Set a function which is called when /sync returns a 'limited' response.
 * It is called with a room ID and returns a boolean. It should return 'true' if the SDK
 * can SAFELY remove events from this room. It may not be safe to remove events if there
@@ -4329,10 +4572,9 @@ function checkTurnServers(client) {
        }
    }, function(err) {
        logger.error("Failed to get TURN URIs");
-        client._checkTurnServersTimeoutID =
-            setTimeout(function() {
- checkTurnServers(client);
-}, 60000);
+        client._checkTurnServersTimeoutID = setTimeout(function() {
+            checkTurnServers(client);
+        }, 60000);
    });
 }

@@ -594,6 +594,11 @@ OlmDevice.prototype.encryptMessage = async function(
        'readwrite', [IndexedDBCryptoStore.STORE_SESSIONS],
        (txn) => {
            this._getSession(theirDeviceIdentityKey, sessionId, txn, (sessionInfo) => {
+                const sessionDesc = sessionInfo.session.describe();
+                console.log(
+                    "Session ID " + sessionId + " to " +
+                    theirDeviceIdentityKey + ": " + sessionDesc,
+                );
                res = sessionInfo.session.encrypt(payloadString);
                this._saveSession(theirDeviceIdentityKey, sessionInfo, txn);
            });
@@ -621,6 +626,11 @@ OlmDevice.prototype.decryptMessage = async function(
        'readwrite', [IndexedDBCryptoStore.STORE_SESSIONS],
        (txn) => {
            this._getSession(theirDeviceIdentityKey, sessionId, txn, (sessionInfo) => {
+                const sessionDesc = sessionInfo.session.describe();
+                console.log(
+                    "Session ID " + sessionId + " to " +
+                    theirDeviceIdentityKey + ": " + sessionDesc,
+                );
                payloadString = sessionInfo.session.decrypt(messageType, ciphertext);
                sessionInfo.lastReceivedMessageTs = Date.now();
                this._saveSession(theirDeviceIdentityKey, sessionInfo, txn);
@@ -23,7 +23,7 @@ limitations under the License.
 */

 import Promise from 'bluebird';
-import logger from '../../../src/logger';
+import logger from '../../logger';

 const utils = require("../../utils");
 const olmlib = require("../olmlib");
@@ -756,6 +756,132 @@ Crypto.prototype.setDeviceVerification = async function(
 };


+function verificationEventHandler(target, userId, roomId, eventId) {
+    return function(event) {
+        // listen for events related to this verification
+        if (event.getRoomId() !== roomId
+            || event.getSender() !== userId) {
+            return;
+        }
+        const content = event.getContent();
+        if (!content["m.relates_to"]) {
+            return;
+        }
+        const relatesTo
+              = content["m.relationship"] || content["m.relates_to"];
+        if (!relatesTo.rel_type
+            || relatesTo.rel_type !== "m.reference"
+            || !relatesTo.event_id
+            || relatesTo.event_id !== eventId) {
+            return;
+        }
+
+        // the event seems to be related to this verification, so pass it on to
+        // the verification handler
+        target.handleEvent(event);
+    };
+}
+
+Crypto.prototype.requestVerificationDM = async function(userId, roomId, methods) {
+    let methodMap;
+    if (methods) {
+        methodMap = new Map();
+        for (const method of methods) {
+            if (typeof method === "string") {
+                methodMap.set(method, defaultVerificationMethods[method]);
+            } else if (method.NAME) {
+                methodMap.set(method.NAME, method);
+            }
+        }
+    } else {
+        methodMap = this._baseApis._crypto._verificationMethods;
+    }
+
+    let eventId = undefined;
+    const listenPromise = new Promise((_resolve, _reject) => {
+        const listener = (event) => {
+            // listen for events related to this verification
+            if (event.getRoomId() !== roomId
+                || event.getSender() !== userId) {
+                return;
+            }
+            const relatesTo = event.getRelation();
+            if (!relatesTo || !relatesTo.rel_type
+                || relatesTo.rel_type !== "m.reference"
+                || !relatesTo.event_id
+                || relatesTo.event_id !== eventId) {
+                return;
+            }
+
+            const content = event.getContent();
+            // the event seems to be related to this verification
+            switch (event.getType()) {
+            case "m.key.verification.start": {
+                const verifier = new (methodMap.get(content.method))(
+                    this._baseApis, userId, content.from_device, eventId,
+                    roomId, event,
+                );
+                verifier.handler = verificationEventHandler(
+                    verifier, userId, roomId, eventId,
+                );
+                // this handler gets removed when the verification finishes
+                // (see the verify method of crypto/verification/Base.js)
+                this._baseApis.on("event", verifier.handler);
+                resolve(verifier);
+                break;
+            }
+            case "m.key.verification.cancel": {
+                reject(event);
+                break;
+            }
+            }
+        };
+        this._baseApis.on("event", listener);
+
+        const resolve = (...args) => {
+            this._baseApis.off("event", listener);
+            _resolve(...args);
+        };
+        const reject = (...args) => {
+            this._baseApis.off("event", listener);
+            _reject(...args);
+        };
+    });
+
+    const res = await this._baseApis.sendEvent(
+        roomId, "m.room.message",
+        {
+            body: this._baseApis.getUserId() + " is requesting to verify " +
+                "your key, but your client does not support in-chat key " +
+                "verification.  You will need to use legacy key " +
+                "verification to verify keys.",
+            msgtype: "m.key.verification.request",
+            to: userId,
+            from_device: this._baseApis.getDeviceId(),
+            methods: [...methodMap.keys()],
+        },
+    );
+    eventId = res.event_id;
+
+    return listenPromise;
+};
+
+Crypto.prototype.acceptVerificationDM = function(event, Method) {
+    if (typeof(Method) === "string") {
+        Method = defaultVerificationMethods[Method];
+    }
+    const content = event.getContent();
+    const verifier = new Method(
+        this._baseApis, event.getSender(), content.from_device, event.getId(),
+        event.getRoomId(),
+    );
+    verifier.handler = verificationEventHandler(
+        verifier, event.getSender(), event.getRoomId(), event.getId(),
+    );
+    this._baseApis.on("event", verifier.handler);
+    return verifier;
+};
+
 Crypto.prototype.requestVerification = function(userId, methods, devices) {
    if (!methods) {
        // .keys() returns an iterator, so we need to explicitly turn it into an array
@@ -803,20 +929,7 @@ Crypto.prototype.beginKeyVerification = function(
        this._verificationTransactions.set(userId, new Map());
    }
    transactionId = transactionId || randomString(32);
-    if (method instanceof Array) {
-        if (method.length !== 2
-            || !this._verificationMethods.has(method[0])
-            || !this._verificationMethods.has(method[1])) {
-            throw newUnknownMethodError();
-        }
-        /*
-        return new TwoPartVerification(
-            this._verificationMethods[method[0]],
-            this._verificationMethods[method[1]],
-            userId, deviceId, transactionId,
-        );
-        */
-    } else if (this._verificationMethods.has(method)) {
+    if (this._verificationMethods.has(method)) {
        const verifier = new (this._verificationMethods.get(method))(
            this._baseApis, userId, deviceId, transactionId,
        );
@@ -951,6 +1064,15 @@ Crypto.prototype.forceDiscardSession = function(roomId) {
 *   the device query is always inhibited as the members are not tracked.
 */
 Crypto.prototype.setRoomEncryption = async function(roomId, config, inhibitDeviceQuery) {
+    // ignore crypto events with no algorithm defined
+    // This will happen if a crypto event is redacted before we fetch the room state
+    // It would otherwise just throw later as an unknown algorithm would, but we may
+    // as well catch this here
+    if (!config.algorithm) {
+        console.log("Ignoring setRoomEncryption with no algorithm");
+        return;
+    }
+
    // if state is being replayed from storage, we might already have a configuration
    // for this room as they are persisted as well.
    // We just need to make sure the algorithm is initialized in this case.
@@ -1817,22 +1939,6 @@ Crypto.prototype._onKeyVerificationStart = function(event) {
            transaction_id: content.transactionId,
        }));
        return;
-    } else if (content.next_method) {
-        if (!this._verificationMethods.has(content.next_method)) {
-            cancel(newUnknownMethodError({
-                transaction_id: content.transactionId,
-            }));
-            return;
-        } else {
-            /* TODO:
-            const verification = new TwoPartVerification(
-                this._verificationMethods[content.method],
-                this._verificationMethods[content.next_method],
-                userId, deviceId,
-            );
-            this.emit(verification.event_type, verification);
-            this.emit(verification.first.event_type, verification);*/
-        }
    } else {
        const verifier = new (this._verificationMethods.get(content.method))(
            this._baseApis, sender, deviceId, content.transaction_id,
@@ -1887,8 +1993,6 @@ Crypto.prototype._onKeyVerificationStart = function(event) {

                    handler.request.resolve(verifier);
                }
-            } else {
-                // FIXME: make sure we're in a two-part verification, and the start matches the second part
            }
        }
        this._baseApis.emit("crypto.verification.start", verifier);
@@ -17,7 +17,7 @@ limitations under the License.
 import Promise from 'bluebird';

 import logger from '../../logger';
-import MemoryCryptoStore from './memory-crypto-store.js';
+import MemoryCryptoStore from './memory-crypto-store';

 /**
 * Internal module. Partial localStorage backed storage for e2e.
@@ -47,42 +47,54 @@ export default class VerificationBase extends EventEmitter {
     *
     * @param {string} transactionId the transaction ID to be used when sending events
     *
-     * @param {object} startEvent the m.key.verification.start event that
+     * @param {string} [roomId] the room to use for verification
+     *
+     * @param {object} [startEvent] the m.key.verification.start event that
     * initiated this verification, if any
     *
-     * @param {object} request the key verification request object related to
+     * @param {object} [request] the key verification request object related to
     * this verification, if any
-     *
-     * @param {object} parent parent verification for this verification, if any
     */
-    constructor(baseApis, userId, deviceId, transactionId, startEvent, request, parent) {
+    constructor(baseApis, userId, deviceId, transactionId, roomId, startEvent, request) {
        super();
        this._baseApis = baseApis;
        this.userId = userId;
        this.deviceId = deviceId;
        this.transactionId = transactionId;
-        this.startEvent = startEvent;
-        this.request = request;
+        if (typeof(roomId) === "string" || roomId instanceof String) {
+            this.roomId = roomId;
+            this.startEvent = startEvent;
+            this.request = request;
+        } else {
+            // if room ID was omitted, but start event and request were not
+            this.startEvent= roomId;
+            this.request = startEvent;
+        }
        this.cancelled = false;
-        this._parent = parent;
        this._done = false;
        this._promise = null;
        this._transactionTimeoutTimer = null;

        // At this point, the verification request was received so start the timeout timer.
        this._resetTimer();
+
+        if (this.roomId) {
+            this._send = this._sendMessage;
+        } else {
+            this._send = this._sendToDevice;
+        }
    }

    _resetTimer() {
-        console.log("Refreshing/starting the verification transaction timeout timer");
+        logger.info("Refreshing/starting the verification transaction timeout timer");
        if (this._transactionTimeoutTimer !== null) {
            clearTimeout(this._transactionTimeoutTimer);
        }
        this._transactionTimeoutTimer = setTimeout(() => {
-           if (!this._done && !this.cancelled) {
-               console.log("Triggering verification timeout");
-               this.cancel(timeoutException);
-           }
+            if (!this._done && !this.cancelled) {
+                logger.info("Triggering verification timeout");
+                this.cancel(timeoutException);
+            }
        }, 10 * 60 * 1000); // 10 minutes
    }

@@ -93,6 +105,8 @@ export default class VerificationBase extends EventEmitter {
        }
    }

+    /* send a message to the other participant, using to-device messages
+     */
    _sendToDevice(type, content) {
        if (this._done) {
            return Promise.reject(new Error("Verification is already done"));
@@ -103,6 +117,21 @@ export default class VerificationBase extends EventEmitter {
        });
    }

+    /* send a message to the other participant, using in-roomm messages
+     */
+    _sendMessage(type, content) {
+        if (this._done) {
+            return Promise.reject(new Error("Verification is already done"));
+        }
+        // FIXME: if MSC1849 decides to use m.relationship instead of
+        // m.relates_to, we should follow suit here
+        content["m.relates_to"] = {
+            rel_type: "m.reference",
+            event_id: this.transactionId,
+        };
+        return this._baseApis.sendEvent(this.roomId, type, content);
+    }
+
    _waitForEvent(type) {
        if (this._done) {
            return Promise.reject(new Error("Verification is already done"));
@@ -140,6 +169,10 @@ export default class VerificationBase extends EventEmitter {
    done() {
        this._endTimer(); // always kill the activity timer
        if (!this._done) {
+            if (this.roomId) {
+                // verification in DM requires a done message
+                this._send("m.key.verification.done", {});
+            }
            this._resolve();
        }
    }
@@ -153,7 +186,7 @@ export default class VerificationBase extends EventEmitter {
                // cancelled by the other user)
                if (e === timeoutException) {
                    const timeoutEvent = newTimeoutError();
-                    this._sendToDevice(timeoutEvent.getType(), timeoutEvent.getContent());
+                    this._send(timeoutEvent.getType(), timeoutEvent.getContent());
                } else if (e instanceof MatrixEvent) {
                    const sender = e.getSender();
                    if (sender !== this.userId) {
@@ -163,9 +196,9 @@ export default class VerificationBase extends EventEmitter {
                            content.reason = content.reason || content.body
                                || "Unknown reason";
                            content.transaction_id = this.transactionId;
-                            this._sendToDevice("m.key.verification.cancel", content);
+                            this._send("m.key.verification.cancel", content);
                        } else {
-                            this._sendToDevice("m.key.verification.cancel", {
+                            this._send("m.key.verification.cancel", {
                                code: "m.unknown",
                                reason: content.body || "Unknown reason",
                                transaction_id: this.transactionId,
@@ -173,7 +206,7 @@ export default class VerificationBase extends EventEmitter {
                        }
                    }
                } else {
-                    this._sendToDevice("m.key.verification.cancel", {
+                    this._send("m.key.verification.cancel", {
                        code: "m.unknown",
                        reason: e.toString(),
                        transaction_id: this.transactionId,
@@ -206,11 +239,17 @@ export default class VerificationBase extends EventEmitter {
            this._resolve = (...args) => {
                this._done = true;
                this._endTimer();
+                if (this.handler) {
+                    this._baseApis.off("event", this.handler);
+                }
                resolve(...args);
            };
            this._reject = (...args) => {
                this._done = true;
                this._endTimer();
+                if (this.handler) {
+                    this._baseApis.off("event", this.handler);
+                }
                reject(...args);
            };
        });
@@ -213,9 +213,10 @@ export default class SAS extends Base {
            message_authentication_codes: MAC_LIST,
            // FIXME: allow app to specify what SAS methods can be used
            short_authentication_string: SAS_LIST,
-            transaction_id: this.transactionId,
        };
-        this._sendToDevice("m.key.verification.start", initialMessage);
+        // NOTE: this._send will modify initialMessage to include the
+        // transaction_id field, or the m.relationship/m.relates_to field
+        this._send("m.key.verification.start", initialMessage);


        let e = await this._waitForEvent("m.key.verification.accept");
@@ -235,7 +236,7 @@ export default class SAS extends Base {
        const hashCommitment = content.commitment;
        const olmSAS = new global.Olm.SAS();
        try {
-            this._sendToDevice("m.key.verification.key", {
+            this._send("m.key.verification.key", {
                key: olmSAS.get_pubkey(),
            });

@@ -306,7 +307,7 @@ export default class SAS extends Base {
        const olmSAS = new global.Olm.SAS();
        try {
            const commitmentStr = olmSAS.get_pubkey() + anotherjson.stringify(content);
-            this._sendToDevice("m.key.verification.accept", {
+            this._send("m.key.verification.accept", {
                key_agreement_protocol: keyAgreement,
                hash: hashMethod,
                message_authentication_code: macMethod,
@@ -320,7 +321,7 @@ export default class SAS extends Base {
            // FIXME: make sure event is properly formed
            content = e.getContent();
            olmSAS.set_their_key(content.key);
-            this._sendToDevice("m.key.verification.key", {
+            this._send("m.key.verification.key", {
                key: olmSAS.get_pubkey(),
            });

@@ -369,7 +370,7 @@ export default class SAS extends Base {
            keyId,
            baseInfo + "KEY_IDS",
        );
-        this._sendToDevice("m.key.verification.mac", { mac, keys });
+        this._send("m.key.verification.mac", { mac, keys });
    }

    async _checkMAC(olmSAS, content, method) {
@@ -1,5 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
+Copyright 2019 The Matrix.org Foundation C.I.C.

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -22,7 +23,7 @@ import Promise from 'bluebird';
 const parseContentType = require('content-type').parse;

 const utils = require("./utils");
-import logger from '../src/logger';
+import logger from './logger';

 // we use our own implementation of setTimeout, so that if we get suspended in
 // the middle of a /sync, we cancel the sync as soon as we awake, rather than
@@ -46,10 +47,15 @@ module.exports.PREFIX_R0 = "/_matrix/client/r0";
 module.exports.PREFIX_UNSTABLE = "/_matrix/client/unstable";

 /**
- * URI path for the identity API
+ * URI path for v1 of the the identity API
 */
 module.exports.PREFIX_IDENTITY_V1 = "/_matrix/identity/api/v1";

+/**
+ * URI path for the v2 identity API
+ */
+module.exports.PREFIX_IDENTITY_V2 = "/_matrix/identity/v2";
+
 /**
 * URI path for the media repo API
 */
@@ -90,6 +96,13 @@ module.exports.MatrixHttpApi = function MatrixHttpApi(event_emitter, opts) {
 };

 module.exports.MatrixHttpApi.prototype = {
+    /**
+     * Sets the baase URL for the identity server
+     * @param {string} url The new base url
+     */
+    setIdBaseUrl: function(url) {
+        this.opts.idBaseUrl = url;
+    },

    /**
     * Get the content repository url with query parameters.
@@ -368,7 +381,18 @@ module.exports.MatrixHttpApi.prototype = {
        return this.uploads;
    },

-    idServerRequest: function(callback, method, path, params, prefix) {
+    idServerRequest: function(
+        callback,
+        method,
+        path,
+        params,
+        prefix,
+        accessToken,
+    ) {
+        if (!this.opts.idBaseUrl) {
+            throw new Error("No Identity Server base URL set");
+        }
+
        const fullUri = this.opts.idBaseUrl + prefix + path;

        if (callback !== undefined && !utils.isFunction(callback)) {
@@ -381,13 +405,17 @@ module.exports.MatrixHttpApi.prototype = {
            uri: fullUri,
            method: method,
            withCredentials: false,
-            json: false,
+            json: true, // we want a JSON response if we can
            _matrix_opts: this.opts,
+            headers: {},
        };
-        if (method == 'GET') {
+        if (method === 'GET') {
            opts.qs = params;
-        } else {
-            opts.form = params;
+        } else if (typeof params === "object") {
+            opts.json = params;
+        }
+        if (accessToken) {
+            opts.headers['Authorization'] = `Bearer ${accessToken}`;
        }

        const defer = Promise.defer();
@@ -395,12 +423,7 @@ module.exports.MatrixHttpApi.prototype = {
            opts,
            requestCallback(defer, callback, this.opts.onlyData),
        );
-        // ID server does not always take JSON, so we can't use requests' 'json'
-        // option as we do with the home server, but it does return JSON, so
-        // parse it manually
-        return defer.promise.then(function(response) {
-            return JSON.parse(response);
-        });
+        return defer.promise;
    },

    /**
@@ -525,76 +548,6 @@ module.exports.MatrixHttpApi.prototype = {
        );
    },

-    /**
-     * Perform an authorised request to the homeserver with a specific path
-     * prefix which overrides the default for this call only. Useful for hitting
-     * different Matrix Client-Server versions.
-     * @param {Function} callback Optional. The callback to invoke on
-     * success/failure. See the promise return values for more information.
-     * @param {string} method The HTTP method e.g. "GET".
-     * @param {string} path The HTTP path <b>after</b> the supplied prefix e.g.
-     * "/createRoom".
-     * @param {Object} queryParams A dict of query params (these will NOT be
-     * urlencoded).
-     * @param {Object} data The HTTP JSON body.
-     * @param {string} prefix The full prefix to use e.g.
-     * "/_matrix/client/v2_alpha".
-     * @param {Number=} localTimeoutMs The maximum amount of time to wait before
-     * timing out the request. If not specified, there is no timeout.
-     * @return {module:client.Promise} Resolves to <code>{data: {Object},
-     * headers: {Object}, code: {Number}}</code>.
-     * If <code>onlyData</code> is set, this will resolve to the <code>data</code>
-     * object only.
-     * @return {module:http-api.MatrixError} Rejects with an error if a problem
-     * occurred. This includes network problems and Matrix-specific error JSON.
-     *
-     * @deprecated prefer authedRequest with opts.prefix
-     */
-    authedRequestWithPrefix: function(callback, method, path, queryParams, data,
-                                      prefix, localTimeoutMs) {
-        return this.authedRequest(
-            callback, method, path, queryParams, data, {
-                localTimeoutMs: localTimeoutMs,
-                prefix: prefix,
-            },
-        );
-    },
-
-    /**
-     * Perform a request to the homeserver without any credentials but with a
-     * specific path prefix which overrides the default for this call only.
-     * Useful for hitting different Matrix Client-Server versions.
-     * @param {Function} callback Optional. The callback to invoke on
-     * success/failure. See the promise return values for more information.
-     * @param {string} method The HTTP method e.g. "GET".
-     * @param {string} path The HTTP path <b>after</b> the supplied prefix e.g.
-     * "/createRoom".
-     * @param {Object} queryParams A dict of query params (these will NOT be
-     * urlencoded).
-     * @param {Object} data The HTTP JSON body.
-     * @param {string} prefix The full prefix to use e.g.
-     * "/_matrix/client/v2_alpha".
-     * @param {Number=} localTimeoutMs The maximum amount of time to wait before
-     * timing out the request. If not specified, there is no timeout.
-     * @return {module:client.Promise} Resolves to <code>{data: {Object},
-     * headers: {Object}, code: {Number}}</code>.
-     * If <code>onlyData</code> is set, this will resolve to the <code>data</code>
-     * object only.
-     * @return {module:http-api.MatrixError} Rejects with an error if a problem
-     * occurred. This includes network problems and Matrix-specific error JSON.
-     *
-     * @deprecated prefer request with opts.prefix
-     */
-    requestWithPrefix: function(callback, method, path, queryParams, data, prefix,
-                                localTimeoutMs) {
-        return this.request(
-            callback, method, path, queryParams, data, {
-                localTimeoutMs: localTimeoutMs,
-                prefix: prefix,
-            },
-        );
-    },
-
    /**
     * Perform a request to an arbitrary URL.
     * @param {Function} callback Optional. The callback to invoke on
@@ -883,7 +836,8 @@ function parseErrorResponse(response, body) {
    let err;
    if (contentType) {
        if (contentType.type === 'application/json') {
-            err = new module.exports.MatrixError(JSON.parse(body));
+            const jsonBody = typeof(body) === 'object' ? body : JSON.parse(body);
+            err = new module.exports.MatrixError(jsonBody);
        } else if (contentType.type === 'text/plain') {
            err = new Error(`Server returned ${httpStatus} error: ${body}`);
        }
@@ -22,7 +22,7 @@ import Promise from 'bluebird';
 const url = require("url");

 const utils = require("./utils");
-import logger from '../src/logger';
+import logger from './logger';

 const EMAIL_STAGE_TYPE = "m.login.email.identity";
 const MSISDN_STAGE_TYPE = "m.login.msisdn";
@@ -174,16 +174,19 @@ InteractiveAuth.prototype = {
            // The email can be validated out-of-band, but we need to provide the
            // creds so the HS can go & check it.
            if (this._emailSid) {
-                const idServerParsedUrl = url.parse(
-                    this._matrixClient.getIdentityServerUrl(),
-                );
+                const creds = {
+                    sid: this._emailSid,
+                    client_secret: this._clientSecret,
+                };
+                if (await this._matrixClient.doesServerRequireIdServerParam()) {
+                    const idServerParsedUrl = url.parse(
+                        this._matrixClient.getIdentityServerUrl(),
+                    );
+                    creds.id_server = idServerParsedUrl.host;
+                }
                authDict = {
                    type: EMAIL_STAGE_TYPE,
-                    threepid_creds: {
-                        sid: this._emailSid,
-                        client_secret: this._clientSecret,
-                        id_server: idServerParsedUrl.host,
-                    },
+                    threepid_creds: creds,
                };
            }
        }
@@ -1,6 +1,7 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
 Copyright 2017 Vector Creations Ltd
+Copyright 2019 The Matrix.org Foundation C.I.C.

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -76,6 +77,7 @@ module.exports.InteractiveAuth = require("./interactive-auth");
 /** The {@link module:auto-discovery|AutoDiscovery} class. */
 module.exports.AutoDiscovery = require("./autodiscovery").AutoDiscovery;

+module.exports.SERVICE_TYPES = require('./service-types').SERVICE_TYPES;

 module.exports.MemoryCryptoStore =
    require("./crypto/store/memory-crypto-store").default;
@@ -21,7 +21,7 @@ const EventEmitter = require("events").EventEmitter;
 const utils = require("../utils");
 const EventTimeline = require("./event-timeline");
 import {EventStatus} from "./event";
-import logger from '../../src/logger';
+import logger from '../logger';
 import Relations from './relations';

 // var DEBUG = false;
@@ -92,6 +92,13 @@ function EventTimelineSet(room, opts) {
 }
 utils.inherits(EventTimelineSet, EventEmitter);

+/**
+ * Get all the timelines in this set
+ * @return {module:models/event-timeline~EventTimeline[]} the timelines in this set
+ */
+EventTimelineSet.prototype.getTimelines = function() {
+    return this._timelines;
+};
 /**
 * Get the filter object this timeline set is filtered on, if any
 * @return {?Filter} the optional filter for this timelineSet
@@ -688,9 +695,12 @@ EventTimelineSet.prototype.compareEventOrdering = function(eventId1, eventId2) {
 * The type of relation involved, such as "m.annotation", "m.reference", "m.replace", etc.
 * @param {String} eventType
 * The relation event's type, such as "m.reaction", etc.
+ * @throws If <code>eventId</code>, <code>relationType</code> or <code>eventType</code>
+ * are not valid.
 *
- * @returns {Relations}
- * A container for relation events.
+ * @returns {?Relations}
+ * A container for relation events or undefined if there are no relation events for
+ * the relationType.
 */
 EventTimelineSet.prototype.getRelationsForEvent = function(
    eventId, relationType, eventType,
@@ -24,7 +24,7 @@ limitations under the License.
 import Promise from 'bluebird';
 import {EventEmitter} from 'events';
 import utils from '../utils.js';
-import logger from '../../src/logger';
+import logger from '../logger';

 /**
 * Enum for event statuses.
@@ -21,7 +21,7 @@ const EventEmitter = require("events").EventEmitter;

 const utils = require("../utils");
 const RoomMember = require("./room-member");
-import logger from '../../src/logger';
+import logger from '../logger';

 // possible statuses for out-of-band member loading
 const OOB_STATUS_NOTSTARTED = 1;
@@ -30,7 +30,7 @@ const ContentRepo = require("../content-repo");
 const EventTimeline = require("./event-timeline");
 const EventTimelineSet = require("./event-timeline-set");

-import logger from '../../src/logger';
+import logger from '../logger';
 import ReEmitter from '../ReEmitter';

 // These constants are used as sane defaults when the homeserver doesn't support
@@ -1057,6 +1057,18 @@ Room.prototype._addLiveEvent = function(event, duplicateStrategy) {
        const redactedEvent = this.getUnfilteredTimelineSet().findEventById(redactId);
        if (redactedEvent) {
            redactedEvent.makeRedacted(event);
+
+            // If this is in the current state, replace it with the redacted version
+            if (redactedEvent.getStateKey()) {
+                const currentStateEvent = this.currentState.getStateEvents(
+                    redactedEvent.getType(),
+                    redactedEvent.getStateKey(),
+                );
+                if (currentStateEvent.getId() === redactedEvent.getId()) {
+                    this.currentState.setStateEvents([redactedEvent]);
+                }
+            }
+
            this.emit("Room.redaction", event, this);

            // TODO: we stash user displaynames (among other things) in
@@ -1175,7 +1187,7 @@ Room.prototype.addPendingEvent = function(event, txnId) {
        for (let i = 0; i < this._timelineSets.length; i++) {
            const timelineSet = this._timelineSets[i];
            if (timelineSet.getFilter()) {
-                if (this._filter.filterRoomTimeline([event]).length) {
+                if (timelineSet.getFilter().filterRoomTimeline([event]).length) {
                    timelineSet.addEventToTimeline(event,
                        timelineSet.getLiveTimeline(), false);
                }
@@ -1204,7 +1216,7 @@ Room.prototype._aggregateNonLiveRelation = function(event) {
    for (let i = 0; i < this._timelineSets.length; i++) {
        const timelineSet = this._timelineSets[i];
        if (timelineSet.getFilter()) {
-            if (this._filter.filterRoomTimeline([event]).length) {
+            if (timelineSet.getFilter().filterRoomTimeline([event]).length) {
                timelineSet.aggregateRelations(event);
            }
        } else {
@@ -43,6 +43,11 @@ const DEFAULT_OVERRIDE_RULES = [
                key: "type",
                pattern: "m.room.tombstone",
            },
+            {
+                kind: "event_match",
+                key: "state_key",
+                pattern: "",
+            },
        ],
        actions: [
            "notify",
@@ -455,6 +460,38 @@ PushProcessor.actionListToActionsObject = function(actionlist) {
    return actionobj;
 };

+/**
+ * Rewrites conditions on a client's push rules to match the defaults
+ * where applicable. Useful for upgrading push rules to more strict
+ * conditions when the server is falling behind on defaults.
+ * @param {object} incomingRules The client's existing push rules
+ * @returns {object} The rewritten rules
+ */
+PushProcessor.rewriteDefaultRules = function(incomingRules) {
+    let newRules = JSON.parse(JSON.stringify(incomingRules)); // deep clone
+
+    // These lines are mostly to make the tests happy. We shouldn't run into these
+    // properties missing in practice.
+    if (!newRules) newRules = {};
+    if (!newRules.global) newRules.global = {};
+    if (!newRules.global.override) newRules.global.override = [];
+
+    // Fix default override rules
+    newRules.global.override = newRules.global.override.map(r => {
+        const defaultRule = DEFAULT_OVERRIDE_RULES.find(d => d.rule_id === r.rule_id);
+        if (!defaultRule) return r;
+
+        // Copy over the actions, default, and conditions. Don't touch the user's
+        // preference.
+        r.default = defaultRule.default;
+        r.conditions = defaultRule.conditions;
+        r.actions = defaultRule.actions;
+        return r;
+    });
+
+    return newRules;
+};
+
 /**
 * @typedef {Object} PushAction
 * @type {Object}
@@ -24,7 +24,7 @@ limitations under the License.
 */

 "use strict";
-import logger from '../src/logger';
+import logger from './logger';

 // we schedule a callback at least this often, to check if we've missed out on
 // some wall-clock time due to being suspended.
@@ -21,7 +21,7 @@ limitations under the License.
 */
 const utils = require("./utils");
 import Promise from 'bluebird';
-import logger from '../src/logger';
+import logger from './logger';

 const DEBUG = false;  // set true to enable console logging.

@@ -0,0 +1,20 @@
+/*
+Copyright 2019 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+export const SERVICE_TYPES = Object.freeze({
+    IS: 'SERVICE_TYPE_IS', // An Identity Service
+    IM: 'SERVICE_TYPE_IM', // An Integration Manager
+});
@@ -19,7 +19,7 @@ import Promise from 'bluebird';
 import SyncAccumulator from "../sync-accumulator";
 import utils from "../utils";
 import * as IndexedDBHelpers from "../indexeddb-helpers";
-import logger from '../../src/logger';
+import logger from '../logger';

 const VERSION = 3;

@@ -16,7 +16,7 @@ limitations under the License.
 */

 import Promise from 'bluebird';
-import logger from '../../src/logger';
+import logger from '../logger';

 /**
 * An IndexedDB store backend where the actual backend sits in a web
@@ -17,7 +17,7 @@ limitations under the License.

 import Promise from 'bluebird';
 import LocalIndexedDBStoreBackend from "./indexeddb-local-backend.js";
-import logger from '../../src/logger';
+import logger from '../logger';

 /**
 * This class lives in the webworker and drives a LocalIndexedDBStoreBackend
@@ -25,7 +25,7 @@ import LocalIndexedDBStoreBackend from "./indexeddb-local-backend.js";
 import RemoteIndexedDBStoreBackend from "./indexeddb-remote-backend.js";
 import User from "../models/user";
 import {MatrixEvent} from "../models/event";
-import logger from '../../src/logger';
+import logger from '../logger';

 /**
 * This is an internal module. See {@link IndexedDBStore} for the public class.
@@ -21,7 +21,7 @@ limitations under the License.
 */

 import utils from "./utils";
-import logger from '../src/logger';
+import logger from './logger';


 /**
@@ -32,7 +32,8 @@ const Group = require('./models/group');
 const utils = require("./utils");
 const Filter = require("./filter");
 const EventTimeline = require("./models/event-timeline");
-import logger from '../src/logger';
+const PushProcessor = require("./pushprocessor");
+import logger from './logger';

 import {InvalidStoreError} from './errors';

@@ -1030,8 +1031,9 @@ SyncApi.prototype._processSyncResponse = async function(
                // honour push rules that were previously cached. Base rules
                // will be updated when we recieve push rules via getPushRules
                // (see SyncApi.prototype.sync) before syncing over the network.
-                if (accountDataEvent.getType() == 'm.push_rules') {
-                    client.pushRules = accountDataEvent.getContent();
+                if (accountDataEvent.getType() === 'm.push_rules') {
+                    const rules = accountDataEvent.getContent();
+                    client.pushRules = PushProcessor.rewriteDefaultRules(rules);
                }
                client.emit("accountData", accountDataEvent);
                return accountDataEvent;
@@ -19,7 +19,7 @@ limitations under the License.

 import Promise from 'bluebird';
 const EventTimeline = require("./models/event-timeline");
-import logger from '../src/logger';
+import logger from './logger';

 /**
 * @private
@@ -1,5 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
+Copyright 2019 The Matrix.org Foundation C.I.C.

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -699,3 +700,11 @@ module.exports.globToRegexp = function(glob, extended) {
    }
    return pat;
 };
+
+module.exports.ensureNoTrailingSlash = function(url) {
+    if (url && url.endsWith("/")) {
+        return url.substr(0, url.length - 1);
+    } else {
+        return url;
+    }
+};
@@ -21,7 +21,7 @@ limitations under the License.
 */
 const utils = require("../utils");
 const EventEmitter = require("events").EventEmitter;
-import logger from '../../src/logger';
+import logger from '../logger';
 const DEBUG = true;  // set true to enable console logging.

 // events: hangup, error(err), replaced(call), state(state, oldState)
@@ -61,9 +61,9 @@ function MatrixCall(opts) {
    this.URL = opts.URL;
    // Array of Objects with urls, username, credential keys
    this.turnServers = opts.turnServers || [];
-    if (this.turnServers.length === 0) {
+    if (this.turnServers.length === 0 && this.client.isFallbackICEServerAllowed()) {
        this.turnServers.push({
-            urls: [MatrixCall.FALLBACK_STUN_SERVER],
+            urls: [MatrixCall.FALLBACK_ICE_SERVER],
        });
    }
    utils.forEach(this.turnServers, function(server) {
@@ -92,8 +92,8 @@ function MatrixCall(opts) {
 }
 /** The length of time a call can be ringing for. */
 MatrixCall.CALL_TIMEOUT_MS = 60000;
-/** The fallback server to use for STUN. */
-MatrixCall.FALLBACK_STUN_SERVER = 'stun:stun.l.google.com:19302';
+/** The fallback ICE server to use for STUN or TURN protocols. */
+MatrixCall.FALLBACK_ICE_SERVER = 'stun:turn.matrix.org';
 /** An error code when the local client failed to create an offer. */
 MatrixCall.ERR_LOCAL_OFFER_FAILED = "local_offer_failed";
 /**
@@ -665,7 +665,7 @@ MatrixCall.prototype._maybeGotUserMediaForAnswer = function(stream) {
        },
    };
    self.peerConn.createAnswer(function(description) {
-        debuglog("Created answer: " + description);
+        debuglog("Created answer: ", description);
        self.peerConn.setLocalDescription(description, function() {
            self._answerContent = {
                version: 0,
@@ -754,7 +754,7 @@ MatrixCall.prototype._receivedAnswer = function(msg) {
 */
 MatrixCall.prototype._gotLocalOffer = function(description) {
    const self = this;
-    debuglog("Created offer: " + description);
+    debuglog("Created offer: ", description);

    if (self.state == 'ended') {
        debuglog("Ignoring newly created offer on call ID " + self.callId +
@@ -1217,24 +1217,9 @@ const _placeCallWithConstraints = function(self, constraints) {
 };

 const _createPeerConnection = function(self) {
-    let servers = self.turnServers;
-    if (self.webRtc.vendor === "mozilla") {
-        // modify turnServers struct to match what mozilla expects.
-        servers = [];
-        for (let i = 0; i < self.turnServers.length; i++) {
-            for (let j = 0; j < self.turnServers[i].urls.length; j++) {
-                servers.push({
-                    url: self.turnServers[i].urls[j],
-                    username: self.turnServers[i].username,
-                    credential: self.turnServers[i].credential,
-                });
-            }
-        }
-    }
-
    const pc = new self.webRtc.RtcPeerConnection({
        iceTransportPolicy: self.forceTURN ? 'relay' : undefined,
-        iceServers: servers,
+        iceServers: self.turnServers,
    });
    pc.oniceconnectionstatechange = hookCallback(self, self._onIceConnectionStateChanged);
    pc.onsignalingstatechange = hookCallback(self, self._onSignallingStateChanged);
@@ -1352,7 +1337,9 @@ module.exports.setVideoInput = function(deviceId) { videoInput = deviceId; };
 * @param {MatrixClient} client The client instance to use.
 * @param {string} roomId The room the call is in.
 * @param {Object?} options DEPRECATED optional options map.
- * @param {boolean} options.forceTURN DEPRECATED whether relay through TURN should be forced. This option is deprecated - use opts.forceTURN when creating the matrix client since it's only possible to set this option on outbound calls.
+ * @param {boolean} options.forceTURN DEPRECATED whether relay through TURN should be
+ * forced. This option is deprecated - use opts.forceTURN when creating the matrix client
+ * since it's only possible to set this option on outbound calls.
 * @return {MatrixCall} the call or null if the browser doesn't support calling.
 */
 module.exports.createNewMatrixCall = function(client, roomId, options) {
@@ -1383,24 +1370,36 @@ module.exports.createNewMatrixCall = function(client, roomId, options) {
            return getUserMedia.apply(w.navigator, arguments);
        };
    }
-    webRtc.RtcPeerConnection = (
-        w.RTCPeerConnection || w.webkitRTCPeerConnection || w.mozRTCPeerConnection
-    );
-    webRtc.RtcSessionDescription = (
-        w.RTCSessionDescription || w.webkitRTCSessionDescription ||
-        w.mozRTCSessionDescription
-    );
-    webRtc.RtcIceCandidate = (
-        w.RTCIceCandidate || w.webkitRTCIceCandidate || w.mozRTCIceCandidate
-    );
-    webRtc.vendor = null;
-    if (w.mozRTCPeerConnection) {
-        webRtc.vendor = "mozilla";
-    } else if (w.webkitRTCPeerConnection) {
-        webRtc.vendor = "webkit";
-    } else if (w.RTCPeerConnection) {
-        webRtc.vendor = "generic";
+
+    // Firefox throws on so little as accessing the RTCPeerConnection when operating in
+    // a secure mode. There's some information at https://bugzilla.mozilla.org/show_bug.cgi?id=1542616
+    // though the concern is that the browser throwing a SecurityError will brick the
+    // client creation process.
+    try {
+        webRtc.RtcPeerConnection = (
+            w.RTCPeerConnection || w.webkitRTCPeerConnection || w.mozRTCPeerConnection
+        );
+        webRtc.RtcSessionDescription = (
+            w.RTCSessionDescription || w.webkitRTCSessionDescription ||
+            w.mozRTCSessionDescription
+        );
+        webRtc.RtcIceCandidate = (
+            w.RTCIceCandidate || w.webkitRTCIceCandidate || w.mozRTCIceCandidate
+        );
+        webRtc.vendor = null;
+        if (w.mozRTCPeerConnection) {
+            webRtc.vendor = "mozilla";
+        } else if (w.webkitRTCPeerConnection) {
+            webRtc.vendor = "webkit";
+        } else if (w.RTCPeerConnection) {
+            webRtc.vendor = "generic";
+        }
+    } catch (e) {
+        logger.error("Failed to set up WebRTC object: possible browser interference?");
+        logger.error(e);
+        return null;
    }
+
    if (!webRtc.RtcIceCandidate || !webRtc.RtcSessionDescription ||
            !webRtc.RtcPeerConnection || !webRtc.getUserMedia) {
        return null; // WebRTC is not supported.