Diff: mbed-mqtt-master/TESTS/mqtt/README.md

Revision:

0:ba7e439238ab

diff -r 000000000000 -r ba7e439238ab mbed-mqtt-master/TESTS/mqtt/README.md
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/mbed-mqtt-master/TESTS/mqtt/README.md	Mon Jun 20 16:24:43 2022 +0000
@@ -0,0 +1,610 @@
+MQTT test plan
+========================
+
+This is a test plan for the Mbed OS MQTT and MQTT-SN protocols. This describes all test cases and their intended behaviors. When an API document is not clear, use this as a reference for implementing correct behavior.
+
+Target API
+----------
+
+The target for this plan is to test (TODO: update links to point to mbed-os master after merge):
+
+-   [legacy MQTTClient paho implementation](../../features/mqtt/paho_mqtt_embedded_c/MQTTClient) with [mbed-os MQTTNetwork implementations](../../mqtt/features/mqtt/src).
+-   [new MQTTClient mbed-os wrapper (both for MQTT and MQTT-SN)](../../mqtt/src/MQTTClientNew.h).
+
+Mbed-os tests are NOT meant to test the underlying Eclipse paho [MQTT](https://github.com/michalpasztamobica/mbed-os/tree/mqtt/features/mqtt/paho_mqtt_embedded_c) and [MQTT-SN](../../features/mqtt/paho_mqtt-sn_embedded_c) libraries. The tests provided are integration tests, checking whether MQTT works correctly with the mbed-os sockets.
+
+Tools to use
+------------
+
+-   Mbed OS.
+-   Standard Mbed OS development tools as described in the [Arm Mbed tools overview](https://os.mbed.com/docs/latest/tools/index.html).
+-   Test server with an MQTT broker.
+
+These test cases themselves do not require any special tooling, other than the MQTT test server described in "Test environment" chapter.
+
+Test environment
+----------------
+
+As a general guideline, network connectivity with public Internet access is required. This satisfies Socket tests, but specific connectivity methods might require some extra configuration or devices within the network.
+
+The test environment consist of DUTs, network connection and an MQTT broker.
+
+### Test server setup
+
+**Required services in the test server**
+
+-   MQTT protcol is enabled on TCP port 1883. Port 8883 for TLS
+-   MQTT-SN protocol is enabled on UDP port 10000.
+
+Configure the firewall to allow this traffic to access the MQTT test server.
+
+Some tests are wrapped with a MQTT_MBED_PUBLIC_TEST_TOPIC_AVAILABLE macro. They will only work if a topic is provided in the broker named "mbed_public_test_topic", that publishes a message with QoS = 0 and content "Test message" with a constant frequency. This is intended for use in testing the subscription without having to publish any additional messages.
+
+**Example configuration for Debian/Ubuntu Linux**
+
+The example test server is using an open-source [Eclipse Mosquitto MQTT broker](https://mosquitto.org/) and [Eclipse paho MQTT-SN Gateway](https://www.eclipse.org/paho/components/mqtt-sn-transparent-gateway/), which are available on many operating systems, and installing them is out of scope of this document. Below is an example of how to install these services into a Debian/Ubuntu based Linux distribution.
+
+***MQTT***
+
+```.sh
+$ sudo apt install mosquitto
+```
+
+The mosquitto broker will start automatically as a deamon on system startup. You can start, stop or restart it using:
+```.sh
+$ sudo service mosquitto [start|stop|restart]
+```
+
+Install the mosquitto clients to be able to test your mosquitto broker locally and generate traffic for a predefined channel:
+
+```.sh
+$ sudo apt install mosquitto-clients
+```
+
+When running the below commands in separate shells:
+
+```.sh
+$ mosquitto_sub -p 1883 -t "testtopic"
+$ mosquitto_pub -p 1883 -t "testtopic" -m "testmessage"
+```
+You should be able to receive the "testmessage" on the subscriber side.
+
+Create a channel that will be fed with messages in a periodic manner. An example of how this can be achieved:
+
+```
+$ watch -n1 mosquitto_pub -p 1883 -m \"Test message\" -t \"mbed_public_test_topic\" -q 0
+```
+
+Make sure QoS is set to 0 (`-q 0`), as this message might be received by multiple devices (or no devices after the tests ends).
+
+Tests which will fail without this functionality are wrapped with a MQTT_MBED_PUBLIC_TEST_TOPIC_AVAILABLE macro.
+
+Optionally edit (or create) the mosquitto broker configuration in /etc/mosquitto/conf.d/ to adjust the port that is used for communication (by default it's 1883):
+
+```
+# Port to use for default listener
+listener 1883
+```
+
+To be able to access the broker from the network you need to open the tcp port that the broker is using:
+```.sh
+$ sudo ufw allow 1883/tcp
+```
+
+Some tests are protected with MQTT_MBED_PASSWORD_PROTECTION_AVAILABLE macro. Those tests can only be run when the MQTT broker is configured to check for a user and password. Follow [this tutorial](http://www.steves-internet-guide.com/mqtt-username-password-example/) to configure the broker.
+
+Make sure that allow_anonymous is set to true, as most test access the broker anonymously.
+
+The tests are by default using user "mbed" and password "1234", so consider setting those on your broker or adjust the tests.
+
+Bear in mind that MQTT user/password is sent in raw text and it has nothing to do with proper TLS encryption described below.
+
+***MQTT TLS***
+
+By default port 8883 is used for TLS-protected MQTT communication.
+
+When the MQTT broker is set up as described above, just follow [this tutorial](http://www.steves-internet-guide.com/mosquitto-tls/) to set up the mqtt broker and all certificates. Use `tls_version tlsv1.2`.
+
+***MQTT-SN***
+
+Once MQTT broker is running it is necessary to set up an MQTT-SN Gateway that would establish a connection to the broker and forward the UDP packets.
+
+Download or clone the [Eclipse paho MQTT-SN implementation](https://github.com/eclipse/paho.mqtt-sn.embedded-c/tree/master/MQTTSNGateway). Follow the guidelines in their repository.
+
+Make sure you adjust the gateway.conf file:
+
+```
+BrokerName=localhost
+```
+
+Also it might be necessary to allow the `PredefinedTopicsList` option in case the Gateway crashes when starting up.
+
+Finally, make sure the Gateway port is emabled for external devices:
+
+```.sh
+$ sudo ufw allow 10000/udp
+```
+
+Providing custom configuration (optional)
+-----------------------------------------
+
+By default the tests will run on a predefined local server as configured in  mbed_lib.json.
+
+It is possible to override the default configuration by providing an `mbed_app.json` file.
+
+The example below will configure the tests to run on a `flespi.io` public MQTT server:
+
+```
+{
+    "config": {
+
+    },
+    "target_overrides": {
+        "*": {
+            "mbed-mqtt.tests-broker-hostname": "\"mqtt.flespi.io\"",
+            "mbed-mqtt.tests-topic": "\"test\"",
+            "mbed-mqtt.tests-public-topic-enable": "false",
+            "mbed-mqtt.tests-public-topic-name": "\"mbed_public_test_topic\"",
+            "mbed-mqtt.tests-username-password-protection-enable": "true",
+            "mbed-mqtt.tests-tls-enable": "false",
+            "mbed-mqtt.tests-username": "\"YOUR_FLESPI_TOKEN\"",
+            "mbed-mqtt.tests-password": "\"\"",
+            "mbed-mqtt.tests-username-always": "false",
+            "mbed-mqtt.tests-mqtt-sn-enable": "false",
+            "target.network-default-interface-type": "ETHERNET"
+           }
+    }
+}
+```
+
+Building test binaries
+--------------------------
+
+For testing the board and driver, test against the Mbed OS master branch to get the most recent, up-to-date test cases and drivers.
+
+To create a build environment:
+
+```.sh
+mbed new network_test
+cd network_test
+cd mbed-os
+git checkout master
+cd ..
+```
+
+MQTT tests will run on any type of connection. See the [relevant section of netsocket tests](../netsocket/README.md#building-test-binaries) to find more information on how to establish connectivity.
+
+Now build test binaries:
+
+```.sh
+mbed test --compile -t <toolchain> -m <target> -n mbed-os-tests-mqtt-mqtt
+```
+
+Running tests
+-------------
+
+When device is connected to network, or in case of wireless device near the access point.
+
+```.sh
+mbed test -n mbed-os-tests-mqtt-mqtt
+```
+
+Test cases for MQTT
+-------------------
+
+The API for legacy MQTT and new MQTT wrapper are identical, so the tests description apply to both of them.
+
+Also the MQTT-SN has the same interface as MQTT, so all "MQTT_*" test also apply to MQTT-SN. 
+ 
+The only difference is in how the Client class is created.
+
+
+### MQTT_CONNECT
+
+**Description:**
+
+Client can connect to broker.
+
+**Preconditions:**
+
+Device is connected to network and connection to MQTT server is established.
+
+**Test steps:**
+
+1. Call `client.connect()` with an valid connectData provided. Check that no error is returned.
+
+**Expected result:**
+
+Call to `client.connect()` succeeds (returns NSAPI_ERROR_OK).
+
+
+### MQTT_CONNECT_NOT_CONNECTED
+
+**Description:**
+
+Client fails to connect when network is not connected.
+
+**Preconditions:**
+
+Device is connected to network.
+
+**Test steps:**
+
+1. Try to connect to a nonexistent MQTT server address or incorrect port.
+2. Call `client.connect()` with a valid connectData provided.
+
+**Expected result:**
+
+Connection fails in step 1. Call to `client.connect()` returns -1 error in step 2.
+
+
+### MQTT_SUBSCRIBE
+
+Client can subscribe to a topic.
+
+**Preconditions:**
+
+Device is connected to network, connection to MQTT broker is established (see the _CONNECT tests).
+
+**Test steps:**
+
+1. Call `client.subscribe()` with a valid topic and QoS0.
+
+**Expected result:**
+
+Call to `client.connect()` succeeds (returns NSAPI_ERROR_OK) in step 1.
+
+
+### MQTT_SUBSCRIBE_NETWORK_NOT_CONNECTED
+
+Client returns an error if subscribing to a topic when there is no connection to MQTT server.
+
+**Preconditions:**
+
+Device is connected to network, but not connected to broker or server.
+
+**Test steps:**
+
+1. Call `client.subscribe()` with a valid topic and QoS0.
+
+**Expected result:**
+
+Call to `client.subscribe()` returns -1 in step 1.
+
+
+### MQTT_SUBSCRIBE_CLIENT_NOT_CONNECTED
+
+Client returns an error if subscribing to a topic when there is no connection to MQTT broker.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT server is established.
+
+**Test steps:**
+
+1. Call `client.connect()` with an invalid connectData provided (for example: empty clientID).
+2. Call `client.subscribe()` with a valid topic and QoS0.
+
+**Expected result:**
+
+Call to `client.subscribe()` returns -1 in step 3.
+
+
+### MQTT_SUBSCRIBE_TOPIC_TOO_LONG
+
+Client returns an error if subscribing to a topic longer than MAX_MQTT_PACKET_SIZE.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+1. Generate a valid string of length larger than MAX_MQTT_PACKET_SIZE.
+2. Call `client.subscribe()` with the too long topic created in step 1 and QoS0.
+
+**Expected result:**
+
+Call to `client.subscribe()` returns -1 in step 2.
+
+
+### MQTT_SUBSCRIBE_INVALID_MESSAGE_HANDLER
+
+Client returns an error if subscribing to a topic without providing a valid message handler.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+1. Call `client.connect()` with an invalid connectData provided (for example: empty clientID).
+2. Call `client.subscribe()` with a valid topic, QoS0 and NULL provided intead of message handler function pointer.
+
+**Expected result:**
+
+Call to `client.subscribe()` returns -1 in step 2.
+
+
+### MQTT_SUBSCRIBE_RECEIVE
+
+Client is able to receive MQTT messages after subscribing to a predefined topic.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+The MQTT broker has a topic "mbed_public_test_topic", which is guaranteed to generate a message before the test times out.
+
+**Test steps:**
+
+1. Call `client.subscribe()` with topic set to "mbed_public_test_topic", QoS0 and a message handler that would prove successful message reception (for example increment a counter).
+2. Wait until a message is received.
+3. Verify that a message was received.
+4. Unsubscribe from the "mbed_public_test_topic" topic.
+
+**Expected result:**
+
+A message is received from the topic in steps 2 and 3. Test does not time out. 
+
+
+### MQTT_UNSUBSCRIBE_WITHOUT_SUBSCRIBE
+
+Client is able to unsubscribe without subscribing to any topic.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+1. Unsubscribe from a valid topic.
+
+**Expected result:**
+
+Call to `client.unsubscribe()` returns NSAPI_ERROR_OK. 
+
+
+### MQTT_UNSUBSCRIBE_INVALID
+
+Client returns an error when invalid topic name is provided.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+1. Unsubscribe from a topic which is an empty string.
+2. Unsubscribe from a topic which has topic of length longer than MAX_MQTT_PACKET_SIZE.
+
+**Expected result:**
+
+Both calls to `client.unsubscribe()` return -1.
+
+### MQTT_PUBLISH
+
+Client is able to publish MQTT messages to a topic.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+1. Call `client.publish()` with a valid topic and message with QoS1.
+2. Call `client.publish()` with a valid topic and message with QoS2.
+3. Call `client.publish()` with a valid topic and message with QoS3.
+
+**Expected result:**
+
+All calls return NSAPI_ERROR_OK.
+
+### MQTT_PUBLISH_NOT_CONNECTED
+
+Client returns an error if publishing to a topic when there is no connection to MQTT broker.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT server is established.
+
+**Test steps:**
+
+1. Call `client.publish()` with a valid topic and QoS0.
+
+**Expected result:**
+
+Call to `client.publish()` returns -1 in step 1.
+
+
+### MQTT_PUBLISH_TOPIC_TOO_LONG
+
+Client returns an error if publishing to a topic with name longer than MAX_MQTT_PACKET_SIZE.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+1. Generate a valid string of length larger than MAX_MQTT_PACKET_SIZE.
+2. Call `client.subscribe()` with the too long topic created in step 1 and QoS0.
+
+**Expected result:**
+
+Call to `client.publish()` returns -1 in step 2.
+
+### MQTT_CONNECT_USER_PASSWORD_INCORRECT
+
+Client returns an error when incorrect password is provided to a password-protected broker.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT server is established.
+
+The broker is password-protected (see set up instructions for details).
+
+**Test steps:**
+
+1. Call `client.connect()` with an incorrect user and password specified.
+
+**Expected result:**
+
+Call to `client.connect()` returns -1.
+
+### MQTT_CONNECT_SUBSCRIBE_PUBLISH
+
+Client is able to subscribe to a topic, publish to it and receive its own messages in a loopback.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT broker is established.
+
+**Test steps:**
+
+All steps should be performed for QoS 0, 1 and 2.
+
+1. Call `client.subscribe()` with a valid topic and register a message handler to verify that messages are recieved (for example with a message counter).
+2. Call `client.publish()` with a previously used topic.
+3. Compare the message counter
+
+**Expected result:**
+
+All calls return no errors and the message is received correctly (for example - message counter increases).
+
+### MQTT_CONNECT_SUBSCRIBE_PUBLISH_USER_PASSWORD
+
+Client is able to provide user and password and subscribe to a topic, publish to it and receive its own messages in a loopback.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT server is established.
+
+The broker is password-protected (see set up instructions for details).
+
+**Test steps:**
+
+All steps should be performed for QoS 0, 1 and 2.
+
+1. Call `client.connect()` with a valid and correct user and password specified.
+2. Call `client.subscribe()` with a valid topic and register a message handler to verify that messages are recieved (for example with a message counter). 
+3. Call `client.publish()` with a previously used topic.
+4. Compare the message counter.
+
+**Expected result:**
+
+All calls return no errors and the message is received correctly (for example - message counter increases).
+
+Test case for MQTT with TLS
+---------------------------
+
+These tests apply only for TLSSocket class both for MQTT legacy API and the new mbed-os wrapper.
+
+In fact MQTT does not provide any TLS support, it uses the underlying TLSSocket to take care of this.
+
+The test case is however relevant in order to verify correct integration of MQTT with TLSSocket and provides a valuable example of how to set up a secure MQTT connection.
+
+### MQTT_TLS_CONNECT_SUBSCRIBE_PUBLISH
+
+Client is able to securely subscribe to a topic, publish to it and receive its own messages in a loopback.
+
+Important: a secure port should be used for this test (by default 8883).
+
+**Preconditions:**
+
+Device is connected to network over a secured socket and a connection to MQTT broker is established.
+
+**Test steps:**
+
+All steps should be performed for QoS 0, 1 and 2.
+
+1. Call `client.subscribe()` with a valid topic and register a message handler to verify that messages are recieved (for example with a message counter).
+2. Call `client.publish()` with a previously used topic.
+3. Compare the message counter
+
+**Expected result:**
+
+All calls return no errors and the message is received correctly (for example - message counter increases).
+
+
+Test cases for MQTT-SN
+----------------------
+
+Most of the MQTT-SN API is identical to MQTT and should follow the tests described earlier. Exceptions:
+
+* MQTT-SN does not provide the user/password functionality so MQTT_CONNECT_USER_PASSWORD_INCORRECT and MQTT_CONNECT_SUBSCRIBE_PUBLISH_USER_PASSWORD cannot be executed for MQTT-SN
+
+The MQTT-SN-specific tests are listed below.
+
+### MQTTSN_IS_CONNECTED
+
+Client can tell that it is connected.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT-SN broker is established.
+
+**Test steps:**
+
+1. Call `client.isConnected()`
+
+**Expected result:**
+
+Call to `client.isConnected()` returns true.
+
+### MQTTSN_IS_CONNECTED_CLIENT_NOT_CONNECTED
+
+Client can tell that it is not connected.
+
+**Preconditions:**
+
+Device is connected to network and a connection to MQTT-SN server (but not broker) is established.
+
+**Test steps:**
+
+1. Call `client.isConnected()`
+
+**Expected result:**
+
+Call to `client.isConnected()` returns false.
+
+### MQTTSN_IS_CONNECTED_NETWORK_NOT_CONNECTED
+
+Client can tell that it is not connected.
+
+**Preconditions:**
+
+Device is connected to network.
+
+**Test steps:**
+
+1. Call `client.isConnected()`
+2. Try to connect client to a non-existent broker.
+3. Call `client.isConnected()`
+
+**Expected result:**
+
+Connection fails in step 2.
+Call to `client.isConnected()` returns false in steps 1 and 3.
+
+### MQTTSN_UDP_CONNECT_SUBSCRIBE_PUBLISH
+
+Client is able to use a UDP Socket to subscribe to a topic, publish to it and receive its own messages in a loopback.
+
+Important: See MQTT-SN section to set up an MQTT-SN gateway for communication. Use an appropriate port (by default 10000).
+
+**Preconditions:**
+
+Device is connected to network over a UDP socket and a connection to MQTT broker is established.
+
+**Test steps:**
+
+All steps should be performed for QoS 0, 1 and 2.
+
+1. Call `client.subscribe()` with a valid topic and register a message handler to verify that messages are recieved (for example with a message counter).
+2. Call `client.publish()` with a previously used topic.
+3. Compare the message counter
+
+**Expected result:**
+
+All calls return no errors and the message is received correctly (for example - message counter increases).