Low Frequency Data

MachineMetrics provides access to your machine data in multiple ways. Aside from our web interface (Operator View and Dashboard), we deliver data in near real time to a AWS Kinesis Stream and an AWS S3 bucket. Follow the steps outlined below to access and consume your data.

Setting up your AWS Account

You will need to have access to an existing AWS Account. For detailed steps, see How to use an external ID when granting access to your AWS resources to a third party.

Please send us your AWS Account ID. We will use the ID to allow the user to access the necessary resources in our account. We will send you the following information back:

ARN of the IAM User to access the resources programmatically (optional)
AWS Access Key ID (optional)
Secret Access Key (optional)
ARN of a role in our account that you need to set up the access policy for the user in your account (optional)
External ID (only if you requested it; see below)
URL to switch roles in the AWS Web Console (optional)
ARN of the Kinesis Stream to access it programmatically or via AWS CLI,
URL of the Kinesis Stream to access in via AWS console,
ARN of the S3 Bucket to access it programmatically or via AWS CLI, and
URL of the S3 Bucket to access in via AWS console.

Allow a User or Role in your Account to Assume the Role in our Account

If necessary, create a AWS User or Role that should have access to the Kinesis Stream and S3 Bucket. Follow detailed steps, see Creating IAM users (console).

To allow a User or Role in your Acc ount to Assume the Role in our account

go to the user’s summary and select the Permissions tab,
click on “add inline policy”, and
select the “JSON” tab.

Paste the following text into the field and fill in the ARN of the Role in our account (you should have already received this information; see above)

Access the Data via the AWS Console

You can view the Kinesis Stream and the S3 Bucket in our account by switching to our role in your AWS Web Console. This is described in the AWS User Guide. A convenient way to do this is to use the URL we have sent you. It starts with https://signin.aws.amazon.com/switchrole .

Once you have switched the role you can access the resources using the direct URLs for the bucket and stream. Because the assumed role is only allowed to view these specific resources, listing all streams or buckets is not possible.

Note: Some browsers do not work well with the way AWS sends downloads. For example, if you download a file using Firefox on a Mac, the browser will automatically unzip the file without renaming it. This might lead to a situation where the file has the suffix “.gz” but is in fact a CSV file. Changing the suffix to “.csv” helps in this case.

Access the Data using the Command Line (AWS CLI)

To install the latest AWS CLI follow the instructions in the AWS User Guide.

Double-check that the AWS CLI uses the credentials that you have allowed to assume the MachineMetrics role:

$ aws sts get-caller-identity

Assume the MachineMetrics role to receive temporary credentials:

$ aws sts assume-role --role-arn "arn:aws:sts::352302322568:assumed-role/the-data-access-role" --role-session-name demo-session
{
    "Credentials": {
        "AccessKeyId": "AXIA***",
        "SecretAccessKey": "ekHn***",
        "SessionToken": "IQoJb3********",
        "Expiration": "2020-09-21T13:12:29+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAVEBW22OEHLCPLSNUM:demo-session",
        "Arn": "arn:aws:sts::352302322568:assumed-role/the-data-access-role/demo-session"
    }
}

Set the credentials to your console (here for Linux or Mac OS):

$ export AWS_ACCESS_KEY_ID=AXIA***
$ export AWS_SECRET_ACCESS_KEY=ekHn***
$ export AWS_SESSION_TOKEN=IQoJb3********

You can now access the objects (files) in the S3 bucket:

$ aws s3 ls <your-bucket-name> --recursive
2020-09-21 06:45:04     223658 timeseries/year=2020/month=09/day=21/hour=10/company-name-2-2020-09-21-10-40-01-014569a8-bfdd-4455-9540-2df0d0a09cd6.gz
2020-09-21 06:50:05     228440 timeseries/year=2020/month=09/day=21/hour=10/company-name-2-2020-09-21-10-45-02-25f191d1-7999-4380-8beb-a95292c4d919.gz
2020-09-21 06:55:06     230399 timeseries/year=2020/month=09/day=21/hour=10/company-name-2-2020-09-21-10-50-03-9eb6065f-8442-49c0-8dd3-9c72d4b6f4ac.gz
2020-09-21 07:00:07     215747 timeseries/year=2020/month=09/day=21/hour=10/company-name-2-2020-09-21-10-55-05-ede94828-3a43-4915-a97c-c1e1fb68127d.gz
2020-09-21 07:05:09     222584 timeseries/year=2020/month=09/day=21/hour=11/company-name-2-2020-09-21-11-00-07-bf7852f6-a864-4084-bc98-615dbe87b090.gz
2020-09-21 07:10:10     202834 timeseries/year=2020/month=09/day=21/hour=11/company-name-2-2020-09-21-11-05-07-c10a608a-7425-4463-8de8-e50274945be3.gz
2020-09-21 07:15:10     201767 timeseries/year=2020/month=09/day=21/hour=11/company-name-2-2020-09-21-11-10-08-35ff289c-6bba-4933-b7e9-cff5f3c04be0.gz
2020-09-21 07:20:11     215434 timeseries/year=2020/month=09/day=21/hour=11/company-name-2-2020-09-21-11-15-08-4cabd16a-6e04-4d31-b885-c35f93e79934.gz

$ aws s3 cp s3://<your-bucket-name>/<file-prefix> .

and the Kinesis Stream:

$ aws kinesis list-shards --stream-name <your-stream-name>
{
    "Shards": [
        {
            "ShardId": "shardId-000000000000",
            "HashKeyRange": {
                "StartingHashKey": "0",
                "EndingHashKey": "340282366920938463463374607431768211455"
            },
            "SequenceNumberRange": {
                "StartingSequenceNumber": "49610314165652095558927892216735774823920940630543433730"
            }
        }
    ]
}

$ aws kinesis get-shard-iterator --stream-name <your-stream-name> --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON
{
    "ShardIterator": "AAAAAAAAAAEW5U0*******"
}

$ aws kinesis get-records --shard-iterator AAAAAAAAAAEW5U0*******
{
    "Records": [
        {
            "SequenceNumber": "49610393841754540869138710484223990110891806681938788354",
            "ApproximateArrivalTimestamp": "2020-09-21T09:03:35.824000-04:00",
            "Data": "MjAyMC0wOS0yMSAxMzowMzozMC45MDcsMjSaMC0wOS0yMSAxMzowMzozNS43MjAsMjczMjZjNjgtMzEyMy00MmZiLWFlZGItMGQ0NjgzNjczYjk5LDcwMTMsU2FtcGxlLExPQUQsWDFsb2FkLDYsLCwsCg=="
        },
        {
            "SequenceNumber": "49610393841754540869138710484225199036711421311113494530",
            "ApproximateArrivalTimestamp": "2020-09-21T09:03:35.827000-04:00",
            "Data": "MjAyMC0wOS0yMSAxMzowMzozMC45MDcsMjCaC0wOS0yMSAxMzowMzozNS43MjAsMjczMjZjNjgtMzEyMy00MmZiLWFlZGItMGQ0NjgzNjczYjk5LDcwMTMsU2FtcGxlLFJPVEFSWV9WRUxPQ0lUWSxTM3NwZWVkLDAsLCwsCg=="
        },
        [...]
    ]
{

The data is Base64 encoded:

$ echo [Data String] | base64 --decode
2020-09-21 13:03:30.907,2020-09-21 13:03:35.720,37326c68-f123-46fb-34fa-0d468f673100,1013,Sample,LOAD,X1load,6,,,,

Access the Data Programmatically (AWS API)

There are many different ways to build applications using the different AWS SDKs. Here are a few links to get you started:

Data Schema

The shape of the data is the same for the Kinesis Stream and the CSV files in the S3 Bucket.

key	description	type
eventTime	the time the machine reported the event	TIMESTAMP
processTime	the time when the event was put into your Kinesis stream	TIMESTAMP
companyId	a UUID unique to each company you have access to	VARCHAR(64)
machineRef	a unique integer value identifying the machine	INTEGER
type	the type of the data value (for example “LOAD”)	VARCHAR(32)
subtype	the subtype of the data value (if hierarchical)	VARCHAR(32)
key	a string identifying the metric (unique within a machine)	VARCHAR(64)
value	the actual value (could be a number or a string, but is always serialized as string)	VARCHAR(512)
nativeCode	the native code of an alarm	VARCHAR(16)
nativeSeverity	the native severity of an alarm	VARCHAR(8)
qualifier	the qualifier for an alarm	VARCHAR(4)
description	a description of an alarm	VARCHAR(64)

Using a Passphrase (ExternalId) to Further Secure Access

The access to your data is protected by encryption and IAM permissions in both accounts. We recommend that you take an additional step if your company allows third parties to assume a role in your account (for example if you are a service provider). The role in the MachineMetrics account that can be assumed by a user in the customer account is not a secret. This can lead to the “confused deputy problem” which is described in more detail in the AWS User Guide . We offer you the option to use a so called ExternalId that can serve as a passphrase every time someone wants to access your data. The downside of this approach is that you cannot access your data through the AWS Web Console, but only through the AWS API or CLI.

Please let our support team know whether you want to use an ExternalId. We will send you the passphrase using a secure channel.

Setting up your AWS Account​

Allow a User or Role in your Account to Assume the Role in our Account​

Access the Data via the AWS Console​

Access the Data using the Command Line (AWS CLI)​

Access the Data Programmatically (AWS API)​

Data Schema​

Using a Passphrase (ExternalId) to Further Secure Access​