2.1.0 Instrument X-Ray

0. Learning Materials

• Video: Week 2 Instrument XRay

• Andrew's repo: week-2-xray

• My branch repo: 02-01-instrument-xray

  • My commits:

    • #9 Instrumented X-ray (and container)

    • #9 Implemented custom x-ray segment

Task List

• Install AWS SDK

• Add X-ray daemon to Docker Compose

  • Add backend variables to Docker Compose

Env variables

Micro service	Variables
backend	HONEYCOMB_API_KEY OTEL_SERVICE_NAME OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS AWS_XRAY_URL AWS_XRAY_DAEMON_ADDRESS
x-ray daemon	AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_REGION

---

1. Workflow

1.1 Install AWS SDK

• Install AWS SDK.

requirements.txt

+ aws-xray-sdk

• Then run:

  • pip install -r requirements.txt

• Create a json file for samling rules.

  • This is necessary because the X-Ray tracing can become expensive very quickly so we are reducing the amount of tgracing analysis by sampling only a limited amount.

cd aws
code xray.json

xray.json

{
    "SamplingRule": {
        "RuleName": "Cruddur",
        "ResourceARN": "*",
        "Priority": 9000,
        "FixedRate": 0.1,
        "ReservoirSize": 5,
        "ServiceName": "backend-flask",
        "ServiceType": "*", 
        "Host": "*",
        "HTTPMethod": "*",
        "URLPath": "*",
        "Version": 1
    }
}

aws xray create-group \
    --group-name "Cruddur" \ 
    --filter-expression "service(\"$FLASK_ADDRESS\" {fault OR error}"

1.2 Add X-ray daemon to DC (Docker Compose)

docker-compose.yml

...
  xray-daemon:
    image: "amazon/aws-xray-daemon"
    environment:
      AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
      AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
      AWS_REGION: "us-east-1"
    command:
      - "xray -o -b xray-daemon:2000"

...

• line 9: run xray on port 2000.

  • -o: local mode. 🍁 By default, X-Ray thinks that it's running on an AWS EC2 instance and tries to grab the metadata from the instance. The -o tag instructs X-ray to not check for EC2 instance metadata.
  • -b: Overrides default UDP address (127.0.0.1:2000).

1.2.1 Add backend variables in DC

Every time we add a container definition to the docker-compose file or we use env variables in the code base, it is important to add the env variables in the docker-compose file.

docker-compose.yml

        AWS_XRAY_URL: "*4567-${GITPOD_WORKSPACE_ID}.${GITPOD_WORKSPACE_CLUSTER_HOST}"
        AWS_XRAY_DAEMON_ADDRESS: "xray-daemon:2000"

• We can commit the implementation at this point.

1.2 Troubleshooting

The Docker container seems to be running. Inside the contianer, I find that the AWS credentials are not correctly injected into the container.

I get no data on X-Ray console. Something must be wrong.

No credentials provided

Hardcoded the credential values into the docker-compose file did the magic (This is a very poor thing to do in terms of security. I'm doing it this way only because I'm testing things out!).

The span data are coming through to X-ray on AWS Cloud.

2. Discussion

2.1 AWS X-Ray

• In our Bootcamp context, this X-ray is another container running alongside our application (the other containers - backend-flask, frontend-react-js, dynamodb-local, and db).

• X-Ray collects the data from our local machine, and send them in batches to AWS using the AWS APIs.

2.2 Middleware

• Middleware is a type of computer software programme that provides services to software applications beyond those available from the operating system. It can be described as "software glue" (Wikipedia).