Using AWS CodeCatalyst with C++
Discovering new tools and platforms that can enhance productivity and streamline workflows is always exciting. Recently, while viewing AWS Conference presentations, I stumbled upon several inspiring videos that introduced me to CodeCatalyst. The enthusiasm and real-world success stories shared by the creators on the platform ignited my curiosity and motivated me to explore how CodeCatalyst could be a game-changer in my own projects. The seamless integrations, user-friendly interface, and powerful features highlighted in these tutorials made me eager to dive in and leverage CodeCatalyst for my future endeavors.
After diving into CodeCatalyst, I quickly realized it didn't offer direct support for C++, which was a bit of a letdown given my interest at the time. But, I soon discovered that its flexible workflow actions could indeed handle C++ builds. Intrigued, I set out on an adventure to see just how much I could harness CodeCatalyst for my projects!
Docker Image
CodeCatalyst utilizes Docker for its build action, which streamlines the process of creating and managing application environments. However, given that the images provided by CodeCatalyst are already quite large and may contain unnecessary components for my specific needs, I opted to construct my own specialized images in a layered manner. This approach not only optimizes the size of the images but also enhances build times. By creating several tailored build images, I can ensure that each image includes only the essential dependencies required for different stages of development, thus improving overall efficiency and resource management in my projects.
Base Image
I began with a basic image that met the minimum requirements for CodeCatalyst, which is designed to facilitate seamless development workflows. This initial image, while functional, did not include any specific language support, limiting its versatility for various programming tasks. My goal was to build upon this foundation, enhancing its capabilities to better align with the needs of diverse development projects.
FROM amazonlinux:latest
# Install packages
RUN yum -y install procps-ng shadow sudo util-linux tar gzip bzip2 unzip less openssh-server openssh-clients git awscli amazon-efs-utils rsync.x86_64 && yum clean all
COPY ./entrypoint.sh /
RUN chmod +x ./entrypoint.sh
# Configure Runtime
RUN ssh-keygen -A && adduser -p '*' mde-user && echo "mde-user ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers.d/mde-user
RUN echo "LANG=C.utf8" >> /etc/environment && echo "LC_ALL=C.utf8" >> /etc/environment && echo "LC_CTYPE=C.utf8" >> /etc/environment
RUN echo "LANG=C.utf8" >> /etc/default/locale && echo "LC_ALL=C.utf8" >> /etc/default/locale && echo "LC_CTYPE=C.utf8" >> /etc/default/locale
RUN echo "LANG=C.utf8" >> /etc/locale.conf && echo "LC_CTYPE=C.utf8" >> /etc/locale.conf
USER mde-user
EXPOSE 22/tcp
EXPOSE 80/tcp
EXPOSE 443/tcp
EXPOSE 8080/tcp
EXPOSE 8443/tcp
CMD ["sleep", "infinity"]
ENV PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
ENTRYPOINT ["sh", "/entrypoint.sh"]
And the entry file from the CodeCatalyst Docker images
#!/bin/sh
set -eu
sudo sh -c '/usr/sbin/sshd -Dd > /var/log/sshd.log 2>&1 &'
exec "$@"
C++ Docker Image
Initially, I created a single Docker image specifically designed for use with C++. This image included the AWS C++ SDK, the C++ AWS Lambda custom runtime library, and all the necessary libraries and build tools essential for my development workflow. The integration of these components aimed to streamline the process of building and deploying C++ applications on AWS. However, after several iterations of using this setup, it became evident that this approach was inefficient. The process of building the AWS SDKs consumed a significant amount of build time with each change, often taking nearly an hour on my system, which severely hindered productivity and slowed down development cycles.
Realizing the need for improvement, I decided to introduce an intermediate image layer. This new layer contains only the basic compiler, essential tools, library includes, and the AWS SDKs, simplifying the overall structure. By layering this beneath the C++ tools image, I aimed to optimize the build process, reduce build times, and enhance overall efficiency without sacrificing the functionality required for my projects. This adjustment allows me to focus more on development rather than waiting for lengthy builds, ultimately leading to a more agile workflow.
C++ Base Image
FROM public.ecr.aws/q6u2b1h2/switched-on-systems/code-catalyst/base:latest
RUN sudo yum install -y gcc-c++ cmake ninja-build openssl-devel libcurl-devel zlib-devel
RUN sudo -- sh -c 'cd /usr/src/ && mkdir awssdk && cd awssdk && \
git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
mkdir build && cd build && \
cmake ../aws-sdk-cpp/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_TESTING=OFF -DAUTORUN_UNIT_TESTS=OFF && cmake --build . -j 8 && \
make install && \
cd /usr/src/ && rm -r awssdk && \
cd /usr/src/ && mkdir lambda-runtime && cd lambda-runtime && \
git clone https://github.com/awslabs/aws-lambda-cpp.git && \
mkdir build && cd build && \
cmake ../aws-lambda-cpp -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local && \
cmake --build . -j 8 && make install && \
cd /usr/src/ && rm -r lambda-runtime'
C++ Tooling Image
FROM public.ecr.aws/q6u2b1h2/switched-on-systems/code-catalyst/cppaws-base:latest
RUN sudo yum install -y wget gdb clang-tools-extra libasan libubsan liblsan libtsan jq libpq-devel gd-devel libpng-devel perl.x86_64 perl-Capture-Tiny.noarch perl-DateTime-Format-DateParse.noarch perl-DateTime.x86_64 perl-JSON.noarch
RUN sudo -- sh -c 'curl -sSL https://github.com/psastras/sarif-rs/releases/download/clang-tidy-sarif-latest/clang-tidy-sarif-x86_64-unknown-linux-gnu -o /usr/bin/clang-tidy-sarif && chmod a+x /usr/bin/clang-tidy-sarif'
RUN sudo -- sh -c 'python3 -m ensurepip --upgrade && python3 -m pip install --upgrade pip'
RUN sudo -- sh -c 'yum -y remove python3-pygments.noarch && pip3 install gcovr'
Build Workflow
From this point forward, I will utilize a sample CMake C++ project as a practical example, gradually building it up step by step to thoroughly demonstrate the process of setting up my C++ CI/CD tooling within CodeCatalyst. This approach will not only showcase the technical aspects but also provide insights into best practices and common pitfalls to avoid along the way.
Kicking things off with an exciting CMake project offers a fantastic opportunity to explore the intricacies of modern C++ development, ensuring a smooth integration with continuous integration and continuous deployment workflows.
cmake_minimum_required(VERSION 3.22)
project(intro_codecatalyst_cpp)
add_executable(intro_codecatalyst_cpp
main.cpp)
target_compile_features(intro_codecatalyst_cpp PRIVATE cxx_std_20)
And an incredibly imaginative code base:
#include
int main() {
std::cout << "Hello CodeCatalyst from C++\n";
return 0;
}
In this code example, we demonstrate how to establish a comprehensive workflow that is triggered specifically by a push to the main branch of our repository. This automated workflow effectively takes the source code from our repository, then configures the CMake build within a directory structure akin to what CLion would utilize, ensuring that the organization of files is optimal for development. After the configuration is complete, the workflow proceeds to compile the code, converting it into executable binaries ready for deployment.
Additionally, it is crucial that we integrate Amazon's Elastic Container Repository (ECR) with the Docker images we have previously created. By doing so, we can efficiently store and manage our Docker images in a secure and scalable environment, facilitating seamless deployment and version control. This integration allows our development process to be more agile and ensures that our applications can be easily distributed across different environments.
Configuration:
# Required - Steps are sequential instructions that run shell commands
Steps:
- Run: mkdir -p cmake-build-release && cd cmake-build-release
- Run: cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -G Ninja
- Run: cmake --build . -j 2
Container:
Image: public.ecr.aws/q6u2b1h2/switched-on-systems/code-catalyst/cpp:latest
Registry: ECR
In the steps of the build action, we begin by creating our build directory, which serves as a dedicated space for all build-related files. This helps keep our project organized and allows for easier management of the build process. After establishing this directory, we change the current working directory to it to ensure that all subsequent commands are executed within this context.
Next, we set up the cmake build system, a powerful tool that automates the generation of build files. During this phase, we configure the necessary parameters and dependencies for our project. Once the setup is complete, we compile the code, which transforms our source files into executable binaries. This process is crucial, as it verifies that our code integrates well and functions as intended.
With this initial setup, we establish a functioning Continuous Integration (CI) and Continuous Deployment (CD) workflow. This workflow not only automates the building and testing of our code but also streamlines the process of integrating changes into the main codebase, thus enhancing collaboration among developers.
While this article will not delve into the deployment aspects of the CI/CD pipeline, it’s important to note that once the build is successfully completed, you would typically export your build artifacts. These artifacts, which include compiled binaries and other necessary files, are essential for use in your deployment actions, ensuring that the final product is ready for end-users.
Dependencies and CodeCatalyst Git authentication
For my projects, I rely on CMake CPM as a powerful source dependency management tool that simplifies the process of integrating third-party libraries into my builds. However, I've encountered a limitation with the build actions in CodeCatalyst, as they are not authorized to access any other repositories within your projects or spaces. To navigate this challenge and ensure smooth operations, I utilize a CI/CD Secret that securely stores the git token associated with my user account. This token is crucial for authenticating my build actions and allowing them the necessary permissions to pull dependencies.
Before executing any build steps, I take the additional precaution of configuring git with an insteadOf action. This setup effectively redirects requests for my libraries, enabling CMake/CPM to access and incorporate any required libraries seamlessly. By implementing this method, I can streamline my workflow and enhance the overall efficiency of my development process.
- Run: git config --global url."https://rolandsos:${Secrets.git-roland-sos}@git.eu-west-1.codecatalyst.aws/".insteadOf "https://git.eu-west-1.codecatalyst.aws/"
This configuration setting is temporary and will last only for the duration of your build action, meaning that it will reset once the process is complete. Fortunately, this temporary setting should not pose any security risks, as it is designed to ensure that your build environment remains stable and secure.
Moreover, you can now take full advantage of CPM's CPMAddPackage, which provides a seamless way to incorporate packages from any GitHub repository. This includes not only third-party libraries but also your own custom libraries that you manage within your various spaces and projects. This versatility allows for greater flexibility and efficiency in your development workflow, enabling you to easily integrate the tools you need to enhance your project.
Unit Testing
Unit testing is a vital part of the software development lifecycle, serving as a foundation for ensuring that individual components of the software function as expected. At its core, unit testing involves writing tests for the smallest testable parts of an application in isolation, typically within the context of a build pipeline. There are various approaches to unit testing, including Behavior-Driven Development (BDD) and Test-Driven Development (TDD). BDD emphasizes collaboration between developers and non-technical stakeholders by creating tests that articulate the desired behavior of a system in plain language. On the other hand, TDD focuses on writing test cases before writing the corresponding code, enforcing a cycle of testing and refactoring. While these methodologies provide structured methods to ensure code quality, this article acknowledges that there are multiple interpretations and implementations of what constitutes a unit test. It does not attempt to prescribe one definitive approach but rather aims to demonstrate the use of CodeCatalyst reporting on unit test and how to achieve them from your C++ project.
Catch2 is a popular C++ unit testing framework known for its simplicity and ease of use. It offers a compact and expressive syntax for writing test cases, making it accessible for developers at all levels. Catch2 includes various test discovery options and integrates smoothly with existing build systems and continuous integration pipelines. With features like section-based testing for organizing complex tests and detailed failure reporting, Catch2 supports comprehensive and maintainable testing practices. The framework encourages a clear and concise expression of test intent, making it an excellent choice for developers aiming to ensure code reliability and quality in their C++ projects.
To enhance the testability of our project, we adopted a strategic approach by dividing our executable into a dedicated library. This library houses all our classes and functions that implement the core business logic, ensuring that our code is modular and organized. We retained only the main.cpp file in the primary executable component, which serves as the entry point for our application. This separation of concerns not only simplifies the structure of our codebase but also facilitates easier updates and maintenance. By linking the library efficiently to both our deliverable artifact and our unit test executable, we can perform comprehensive testing and quality assurance, leading to more reliable software outcomes. This methodology ultimately fosters better collaboration among team members and enhances our overall development workflow.
We use CPM to add the Catch2 testing framework to our project and link it against the our unit test executable:
CPMAddPackage("gh:catchorg/Catch2@3.5.2")
target_link_libraries(unit_tests PRIVATE intro_codecatalyst_lib Catch2::Catch2WithMain)
Similar to the code coverage report, we incorporate unit tests as an additional build step to generate the report file, which provides valuable insights into the effectiveness of our testing efforts. After thoroughly evaluating various output formats, we determined that the JUnit output format of Catch2 integrates effectively with CodeCatalyst's parser designed specifically for that format. This integration not only streamlines our workflow but also enhances our ability to track and analyze test results, ensuring that we maintain high-quality code throughout the development process. By leveraging this approach, we can identify potential issues early and mitigate risks, ultimately leading to more robust and reliable software.
./unit_tests -r junit::out=unit-tests.xml -r console::out=-::colour-mode=ansi [main]
And finally, we add the comprehensive report to our output artifacts, which allows us to conveniently view and analyze it in CodeCatalyst, ensuring that all relevant data and insights are easily accessible for our review and decision-making process.
Unit-Tests:
SuccessCriteria:
PassRate: 100
IncludePaths:
- cmake-build-release/unit-tests.xml
Format: JUNITXML
As we increasingly utilize Behavior Driven Development (BDD) for our tests, we have taken a significant step forward by developing a custom report writer. This tool optimizes the JUnit format, allowing us to generate well-structured and insightful reports on the CodeCatalyst Console, which enhances our testing efficiency and clarity. The reports provide not only test results but also valuable insights into the behavior of the code, making it easier for our team to identify areas for improvement.
We sincerely appreciate the outstanding support from the CodeCatalyst team in addressing various challenges related to parsing and interpreting the reports. Their expertise and prompt assistance have been instrumental in helping us fine-tune our reporting process, ultimately enabling us to implement an effective solution that meets our needs.
A special thank you is extended to their developers and the entire CodeCatalyst team for their hard work and dedication. Their contributions have made a significant impact on our workflow, and we look forward to continuing this collaborative journey to further enhance our testing capabilities.
Static Analyzer
CodeCatalyst offers robust support for comprehensive code quality reports, encompassing those generated by static analyzers in various formats to ensure a thorough assessment of your codebase. Among these formats, we have been extensively utilizing the SARIF (Static Analysis Results Interchange Format), which is particularly beneficial for sharing and analyzing static analysis results across different tools and platforms. This format can be conveniently generated using clang-tidy-sarif, a tool derived from the clang-tidy static analyzer, which helps developers identify potential issues in their code early in the development process, fostering better coding practices and enhancing overall code quality.
Our CMake toolchain provides a user-friendly way to activate Clang-Tidy, a powerful static analysis tool, with just a simple configuration option. By enabling this feature, developers can run the static analyzer concurrently with the compilation of their code, allowing them to catch potential issues early in the development process. This integration not only enhances code quality but also streamlines the workflow, making it easier for teams to adhere to coding standards and best practices while improving overall productivity.
set(CLANG_TIDY_CHECKS -*,bugprone-*,clang-analyzer-*,concurrency-*,cppcoreguidelines-*,performance-*,misc-*)
set(CMAKE_CXX_CLANG_TIDY clang-tidy;-checks=${CLANG_TIDY_CHECKS})
In the first line, we specify the tests and checks to be included for comprehensive code analysis, ensuring that our code meets quality and performance standards. For guidance on selecting your preferred checks, you may refer to the clang-tidy documentation, which provides a detailed overview of available checks, along with explanations of their purposes and usage. The checks presented here are those that we commonly utilize in our projects to catch potential issues early. Additionally, you may want to explore the modernizer section, which assists in identifying code that requires updates to align with current C++ coding standards, helping maintain code quality and readability over time. If you prefer a more tailored approach rather than using entire sections, you can individually list the specific checks you wish to run or exclude particular checks from the end of the list, offering flexibility in how you manage code quality. One check that we typically exclude for unit testing is the CPP Core Guidelines check concerning magic numbers, as we focus on ensuring that our tests remain clear and maintainable. The second line of the configuration instructs cmake to enable clang-tidy during the build process, integrating this powerful static analysis tool seamlessly into our workflow to enhance code quality continuously.
We can now efficiently direct our compiler output and standard error into clang-tidy-sarif, a tool designed to convert static analysis log output into a structured format. This process enables us to generate the necessary report format that CodeCatalyst can read and present our results in a user-friendly way, making it easier for developers to understand and act on the insights provided.
cmake --build . -j 2 2>&1 | clang-tidy-sarif
To ensure our compiler output remains accessible for future reference in the pipeline's log, we utilize the `tee` command to effectively redirect the compiler output into a designated file. This approach not only captures the output in real-time but also allows us to retain a permanent record of the compilation process. Subsequently, in the next build step, this captured output file is transformed into a SARIF report, which provides structured information about the compiler's findings and enhances our ability to analyze and address any issues that may arise during development. This two-step process helps maintain clarity and thoroughness in our build and debugging workflows.
cmake --build . -j 2 2>&1 | tee build.log
cat build.log | clang-tidy-sarif > static-analyzer.sarif
In the final step of our process, we will have CodeCatalyst incorporate our SARIF static analysis report by adding it to our workflow configuration. This integration will allow us to streamline our development pipeline, enabling continuous monitoring of code quality and ensuring that any potential issues identified in the report are addressed promptly. By embedding the SARIF report into our workflow, we can enhance our overall code review process and maintain high standards of software reliability.
Reports:
Static-Analysis:
SuccessCriteria:
StaticAnalysisFinding:
Severity: CRITICAL
Number: 0
IncludePaths:
- cmake-build-release/static-analyzer.sarif
Format: SARIFSA
Lastly, upon executing our build pipeline, we receive a detailed visual report generated by the static analyzer. This report includes an overview of the analysis process, highlighting key metrics and trends that can help us gauge the overall health of our codebase.
The summary, along with a comprehensive list of all findings, is readily accessible from the initial screen. This user-friendly interface allows us to delve deeper into each individual source file, where we can thoroughly review our code, complete with annotations that provide insights and context from the static analyzer's findings. This process not only helps in identifying potential issues but also promotes best practices in coding, ultimately leading to a more robust and maintainable application.
Code Coverage
Code coverage is a crucial metric in software development that measures the extent to which source code is executed during testing. Its primary objective is to ensure that a significant portion of the codebase is tested, thereby enhancing software quality and reliability. However, relying solely on code coverage as a measure of developers' productivity can be misleading and counterproductive. High code coverage numbers may appear impressive, but they do not guarantee the absence of bugs or the logical correctness of the code. Furthermore, a focus on increasing code coverage percentages might lead developers to write superficial tests that do not thoroughly validate code functionality. This misuse can shift focus away from creating meaningful and effective tests that genuinely improve the codebase, leading to a false sense of quality assurance while potentially overlooking complex edge cases or critical pathways in the application. Therefore, while code coverage is a valuable tool, it should be used judiciously within a broader context that considers other quality metrics and practices.
To enable code coverage reporting for our project, we begin by meticulously compiling and linking our code with the necessary flags specified by our compiler. This process is crucial as it ensures that we gather detailed insights into which parts of our code are being executed during testing. The example provided here specifically pertains to the GCC toolchain, which is widely used due to its efficiency and robustness in handling various programming languages.
In this straightforward illustration, we focus on applying coverage generation solely to the main executable. This choice simplifies the example, allowing us to clearly demonstrate the process without overwhelming details. However, in a more realistic scenario, one would typically utilize the coverage results generated from running the unit test executable. By doing so, we can achieve a more comprehensive understanding of our code's performance and identify areas that require further optimization or testing. This approach not only enhances the quality of our software but also contributes to more efficient debugging and development practices in the long run.
target_compile_options(intro_codecatalyst_cpp PRIVATE --coverage -fno-omit-frame-pointer)
target_link_options(intro_codecatalyst_cpp PRIVATE --coverage -fno-omit-frame-pointer)
In our build process we than run our executable to generate the code coverage report
cd ../ && gcovr -e cmake-build-release/_deps/ --cobertura cmake-build-release/code-coverage.xml
And finally add our report to our CodeCatalyst Workflow
Code-Coverage:
Format: COBERTURAXML
IncludePaths:
- cmake-build-release/code-coverage.xml
The Code Coverage Overview, like the other report overviews, gives a quick summary of your project's code coverage.
From the Overview you can dig deeper into assessing each file and branch coverage.
And more
This little introduction into using the CodeCatalyst tooling with C++ is just the start of what you can do and I hope it will be helpful to anyone trying to do similar work.
All our test projects are being built in separate and parallel branches for a production build and a testing build. The testing builds include sanitizer options so anything missed by the static analyzer might get picked up at runtime.
And of course our aim is not only to produce a build artifact, but to deliver a product and we make good use of the AWS deployment infrastructure, such as CloudFormation, to deliver out applications into development and production environments in a reliable and repeatable way.
We also use the docker images that are used on CodeCatalyst for local builds in the same runtime environment as the final runtime in AWS. This integrates nicely into the AWS SAM tooling during development to quickly deploy artifacts from the local machine.
Conclusion
By harnessing the power of tools like CodeCatalyst alongside complementary AWS services, such as CloudFormation and SAM, we are not only streamlining our development processes but also elevating the standard of our software. These tools enable us to test thoroughly, deploy confidently, and iterate swiftly, leveraging automated and reliable workflows that minimize human error and maximize efficiency. As we continue to integrate these technologies into our pipeline, we anticipate a boost in our ability to deliver robust, high-quality software that consistently meets the needs of our users. The strategic use of these tools is essential for any development team aiming to maintain a competitive edge in today's fast-paced tech landscape.