SKA Mid.CBF Engineering Console

Documentation on the Developer’s portal: ReadTheDocs

Code repository: ska-mid-cbf-engineering-console

The Engineering Console is being built in a Docker container, which insulates it from variations in the server environment. In addition to enabling MCS-Talon integration and testing, this container can be used to provide a controlled environment for automated end-to-end Talon HW testing.

The Engineering Console Docker is built in the pipeline and deployed to the Central Artefact Repository CAR.

Installation

git clone https://gitlab.com/ska-telescope/ska-mid-cbf-engineering-console
cd ska-mid-cbf-engineering-console
git submodule init
git submodule update
poetry install # to create the virtual environment with all dependencies
poetry shell # to run shell in the virtual environment
make oci-build # or "poetry run make oci-build" if not in the poetry shell
make run    # runs "hello world" test

Usage

Run the Docker interactively

To run the docker interactively:

make run-interactive

which opens a bash shell in the docker. To see available test script options:

./talondx.py --help

To install vim/nano editors while in interactive mode:

apt-get update
apt-get -y install vim nano

Generate Talondx Config FIle

To auto-generate the talondx config file based on the board configuration. Run the following command:

make generate-talondx-config BOARDS="<BOARDS>"

where is a comma-delimited list ofboard numbers you wish to turn on and deploy the HPS device servers onto. Example “1,2,3” if you wish to turn on and run device servers on talon1, talon2, and talon3.

Download Artefacts from CAR

To download FPGA bitstreams and Talon Tango device binaries from CAR to the local folder specified in the Makefile (TALONDX_DEST_DIR):

make download-artifacts

or specify a different destination folder:

make download-artifacts TALONDX_DEST_DIR="destination-folder"

A different config JSON can be specified if it exists as well (default value in the Makefile);

make download-artifacts

To upload new FPGA bitstreams to CAR for use, see the ska-mid-cbf-talondx project

Optional: Override DS Artefacts with local build

In order for this script to work, ensure to clone and build your device servers in the same root directory: Example: If clone ds-vcc and ds-lstv-gen device servers ensure both are cloned under the same directory which would like like:

/home/user/dev/ds/ds-lstv-gen
/home/user/dev/ds/ds-vcc

To override the device servers (ds-lstv-gen,ds-vcc in this example) run the following command:

make ds_list=ds-lstv-gen,ds-vcc ds_basedir=<path to ds base directory> ec_dir=<path to ec checkout> ds-override-local

where ds_basedir is the path to the device server root directory of clone, /home/user/dev/ds from the previous example

Update the Tango DB inside MCS

make config-db

This command adds the Talon device servers as specified in the talondx-config.json file.

Note: the artefacts need to be downloaded before updating the database (the artefacts contain detri JSON files needed for the DB update).

Pull and run the Docker from CAR

docker pull artefact.skao.int/ska-mid-cbf-engineering-console:0.0.2
docker run artefact.skao.int/ska-mid-cbf-engineering-console:0.0.2

Read the Docs

The Engineering Console project auto-generates Read the Docs documentation, which includes this README.

To re-generate the documentation locally prior to checking in updates to Git:

make documentation

To see the generated documentation, open /ska-mid-cbf-engineering-console/docs/build/html/index.html in a browser – e.g.,

firefox docs/build/html/index.html &

Configure the Talon-DX Boards from MCS

The Talon DX boards can be configured with binaries from CAR using a combination of Engineering Console and MCS both running the Dell Server – see MCS-Talon+Integration for details.

How to Run the On Command Sequence

1. Install MCS and Engineering Console

Install MCS, then Install Engineering Console.

2. Download Artefacts

Follow the instructions in Download Artefacts from CAR

3. Start MCS

Follow the instructions in MCS up to the make install-chart step to get MCS running.

4. Configure Tango DB

Follow the steps in Update the Tango DB inside MCS.

5. Ensure that Talon DX Boards are Off

Run the commands on either the Dell1 or Dell2 servers:

To check the current power status:

/shared/talon-dx-utilities/bin/talon_power_lru.sh <lru>

If the boards are powered on, power off the boards:

/shared/talon-dx-utilities/bin/talon_power_lru.sh <lru> off

Where is the lru associated to the board you wish you run your tests on. As of now, lru1 is associated to talon1/talon2 and lru2 is associated to talon3/talon4.

6. Send On Command from MCS

Run the required MCS On command sequence using:

make mcs-on SIM=<Simulation Mode>

SIM=1 if you want to run in Simulation Mode for all MCS devices. SIM=0 if you want to run without simulation mode and target the HPS devices on the talons.

7. Read Talon HPS Device Version and Status Information

To do a quick check that all devices turned on with the mcs-on command:

make device-check

To display Talon DS version information (version, build date, Git commit hash):

make talon-version

To repeatedly display the current Talon DS state and status:

make talon-status

Configure an end-to-end scan to generate visibilities

This section assumes that you have followed all the steps in the previous section to power on and configure the Talon boards. At this point the three MCS devices that were configured in the previous step should be in the ON state. To perform a single receptor basic correlation, use the following command:

make basic-correlation BOARDS=<target talon>

Run a visibility capture

This section assumes the previous basic-correlation section has been successfully completed.

Run make command to start a visibility capture. Right after running the command the directory path will be printed as to where the data will be located if the command completes successfully.

make visibility-capture SV_VER=<Signal Verification Target Version>

While visibility capture is running, in another window issue the scan command to MCS

make mcs-scan BOARDS=<target talon>

After some time, issue the end scan command.

make mcs-end-scan BOARDS=<target talon>

Run a SLIM Mesh test

This section assumes that the talondx config file has been generated and the tango-db has been configured. Additionally, the talon boards in use must be running ds-slim-tx-rx device servers, either as standalone binaries or via MCS.

The slim_mesh_config.yaml file contains the full list of frequency slices to be supported by AA1 and their associated SLIM links. For AA0.5, only a subset of these links need be supported. As such, a trailing [x] indicates that a given link will be ignored in the mesh test. This allows links to be toggled on and off as the project scales.

SLIM links support a serial loopback mode, which internally directs the Tx data to its Rx pair without using the optical transceiver (eg. Tx0->Rx0). Serial loopback can be enabled globally by passing a 1 to the LOOPBACK argument. By default, serial loopback is disabled.

With the YAML file configured, the deployer image needs to be rebuilt. Then the test can be run with the following command:

make oci-build OCI_IMAGE=ska-mid-cbf-engineering-console-deployer
make mesh-test LOOPBACK=<SERIAL_LOOPBACK_MODE>

Notes on Dish Packet Capture

1. On Command Sequence

Follow How to Run the On Command Sequence

3. Set up BITE devices

From the root directory of the engineering console run the following:

make talon-bite-config BOARDS="<BOARDS>"

where is a comma-delimited list of board numbers on which you wish to configure the BITE device servers. Example “1,2,3” if you wish to configure the BITE device servers on talon1, talon2, and talon3.

NOTE: Only the talon board defined in TALON_UNDER_TEST will be configured with the destination MAC address therefore only data from this board will reach the destination interfaces.

4. Dish Packet Capture

Open a new terminal. From the root directory of the engineering console run the following:

make dish-packet-capture

This command will not exit until the following command is run.

5. Start LSTV Replay

From the root directory of the engineering console run the following:

make talon-bite-lstv-replay BOARDS="<BOARDS>"

where is a comma-delimited list of board numbers on which you wish to replay data from. Example “1,2,3” if you wish to replay data from talon1, talon2, and talon3.

Notes on Signal Chain Verification

Wideband State Count

Collect WB state count histogram and power spectrum vectors

make wb-state-count-capture

Generate a report from the WB state count vectors collected by the previous command.

make wb-state-count-report

Set WB_STATE_COUNT_LOCAL_DIR to specify the directory to store the outputs. By default this is ./mnt/wb-state-count

Notes on MCS Interfaces

Commands

MCS commands can additionally be sent from Taranta (previously known as Webjive) or the itango3 shell.

Send the On command to CBF Controller from Taranta

Taranta needs to be enabled in MCS – see Taranta instructions for details.

Send commands to CBF Controller from itango3 shell

$ kubectl exec -it cbfcontroller-controller-0 -n ska-mid-cbf -- itango3
Defaulted container "device-server" out of: device-server, wait-for-configuration (init), check-dependencies-0 (init), check-dependencies-1 (init), check-dependencies-2 (init), check-dependencies-3 (init), check-dependencies-4 (init), check-dependencies-5 (init), check-dependencies-6 (init), check-dependencies-7 (init)
ITango 9.3.3 -- An interactive Tango client.

Running on top of Python 3.7.3, IPython 7.21 and PyTango 9.3.3

help      -> ITango's help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

IPython profile: tango

hint: Try typing: mydev = Device("<tab>

In [1]: cbf_controller = DeviceProxy("mid_csp_cbf/sub_elt/controller")

In [2]: cbf_controller.State()
Out[2]: tango._tango.DevState.ON

In [3]: cbf_controller.Status()
Out[3]: 'The device is in OFF state.'

In [4]: cbf_controller.On()
Out[4]: [array([0], dtype=int32), ['On command completed OK']]

In [5]: cbf_controller.Status()
Out[5]: 'The device is in OFF state.'

In [6]: cbf_controller.State()
Out[6]: tango._tango.DevState.ON

Send ConfigureScan command from itango3 shell

In [1]: controller = DeviceProxy("mid_csp_cbf/sub_elt/controller")

In [2]: subarray = DeviceProxy("mid_csp_cbf/sub_elt/subarray_01")

In [3]: controller.On()
Out[3]: [array([0], dtype=int32), ['On command completed OK']]

In [4]: subarray.AddReceptors(["MKT000", "MKT001", "MKT002", "MKT003"])
Out[4]: [array([0], dtype=int32), ['CBFSubarray AddReceptors command completed OK']]

In [5]: f = open("tests/data/ConfigureScan_basic.json")

In [6]: subarray.ConfigureScan(f.read().replace("\n", ""))
Out[6]: [array([0], dtype=int32), ['CBFSubarray Configure command completed OK']]

or paste the following into the itango3 shell:

controller = DeviceProxy("mid_csp_cbf/sub_elt/controller")
subarray = DeviceProxy("mid_csp_cbf/sub_elt/subarray_01")
controller.On()
subarray.AddReceptors(["MKT000", "MKT001", "MKT002", "MKT003"])
f = open("tests/data/ConfigureScan_basic.json")
subarray.ConfigureScan(f.read().replace("\n", ""))

Note: the test file tests/data/ConfigureScan_basic.json is part of the MCS codebase and is available when connected using itango3.

Send Scan command to VCC from itango3 shell

vcc = DeviceProxy("mid_csp_cbf/vcc/002")
vcc.simulationMode = 0
vcc.adminMode = 0
vcc.On()

vcc.ConfigureBand("1") # Only bands 1 and 2 are supported by the HPS software
vcc.ConfigureScan("{\
    \"config_id\": \"test_config\",\
    \"frequency_band\": \"1\",\
    \"frequency_band_offset_stream_1\": 5,\
    \"frequency_band_offset_stream_2\": 0,\
    \"rfi_flagging_mask\": "",\
    \"fsp\": [\
        {\
            \"fsp_id\": 1,\
            \"frequency_slice_id\": 3,\
            \"function_mode\": \"CORR\"\
        }\
    ]\
}") # This is an example of the expected argument format for the VCC

vcc.Scan("6") # Use any arbitrary integer ID

vcc.EndScan()

Logs

View logs from a single MCS pod in the terminal

To see the CBF controller logs:

kubectl logs -f cbfcontroller-controller-0 -n ska-mid-cbf

where cbfcontroller-controller-0 is the pod name shown when running make watch in MCS.

View logs using K9S

k9s -n ska-mid-cbf

then select the pod (e.g., cbfcontroller-controller-0) and press l to view the logs.

Notes

Raw Repository

FPGA bitstreams are uploaded manually to the raw repository in CAR (Common Artefact Repository, https://artefact.skatelescope.org/) here:

raw-internal/ska-mid-cbf-talondx/fpga-test/talon_dx-{_bitstream-name_}-v{_version_}.tar.gz

Example - manually package the BITE bitstream files

mkdir bin
cp bite5.json bin/
cp mvp5_wip02.core.rbf bin/
cp mvp5_wip02.dtb bin/
tar -cvf talon_dx-bite-v0.5.0.tar bin
gzip -k talon_dx-bite-v0.5.0.tar

Example - manually unpackage the BITE bitstream files

gzip -d talon_dx-bite-v0.5.0.tar.gz
tar -xvf talon_dx-bite-v0.5.0.tar

where {version} is in the X.Y.Z format.