Automatisch

Improving Software Quality and Testing Infrastructure for Automatisch

We recently collaborated with an open source no-code automation company called Automatisch and helped them both improve their product with our numerous QA services. This includes implementing automated tests, bug finding and reporting, and enhancing their continuous integration (CI) pipeline to support both community and enterprise editions.

Automatisch: The Open Source Zapier Alternative

Automatisch is a reputable open source software integration platform providing an alternative to Zapier. It is built with a variety of node JS libraries, including ReactJS and expressJS. Upon starting our collaboration, their product already had several dozen integrations with other platforms, and were in progress building many more. The key quality assurance (QA) requirements included discovering and documenting bugs, developing automated smoke tests using playwright, and ensuring the CI pipeline works flawlessly for both community and enterprise editions.

Proactive Bug Finding and Reporting

Our approach to reporting bugs

While working through our testing of Automatisch's platform, we discovered bugs covering a wide span of cases. All of our bug reports included three sections

• Overview: A Quick overview of the bug found, which generally includes an image
• Reproduction: Detailed list of instructions for reproducing the bug
• Improvements: Suggestions for how to resolve the bug and improve the software

Using this format makes it clear for future developers where they can find the source of the problem and resolve the ticket.

Some Bugs We Found

Recreating deleted users: If an admin deleted a user they manage, and then attempted to re-create that user with the same email address, they were met with not only an opaque error message, but also did not have a way to recover the user's old account.

Inconsistent interface conveying permissions: If a user does not have permissions to create a connection to a platform integrated into Automatisch, there was inconsistent behavior in the interface depending on the page they accessed. For example, the user could

Deleting roles edge case: If a only a deleted user has a role attached to them, the admin is not able to delete the role after the user has been deleted.

Invalidating GitHub OAuth error: If a user invalidates the oauth token used by Automatisch, there is not a useful error reported, only a generic error message which could imdicate many different problems.

Dive into Testing

Upon starting out work with Automatisch, we dove into a codebase already containing tests for some of the features their SaaS provides. One of the first tasks we worked on was refactoring one of the main smoke tests into one which followed best practices in Playwright.

Refactoring Workflow Tests

The central workflow configuration interface in Automatisch is their bread and butter: this is where users go to configure their custom integrations between the various supported platforms. The set of tests for this part of the codebase included some large multi-part test cases which needed to be run sequentially. We refactored this set of code using Playwright's test.step interface, which makes it easy to write a large multipart test into smaller easily comprehendable test units. By refactoring we ensured the test ordering was guaranteed to be stable across platforms, ensuring replicability of test runs while also giving assurances for working featres across software releases.

User Management

After this refactoring, we began work writing tests for other parts of the application. This included writing test cases for managing users for the enterprise-edition of Automatisch. Part of writing the tests is setting up the application code so it is compatible with the tests, even after user interface changes.

Our tests for user management captured a wide number of workflows administrators could face while using Automatisch. This includes the basic user workflows, such as creating users from the administrators side, editing the users, and finally deleting them. In addition to the standard checks, we also wrote tests ensuring coverage of edge case behavior. Note that when a user accesses Automatisch, they must provide their email address as a username.

We included tests in the administrator workflow ensuring a second user cannot be created with the same email address as an existing user. In addition, there are tests ensuring that if an administrator attempts to edit one user to have the same email as another existing user, this action cannot be performed as well.

One of the challenges we ran into while writing these tests was ensuring the notifications plugin was not interfering with test behavior. After performing an action, a notification in the application would be displayed showing whether or not the action was successful. For example, if we tried editing the information of a user, either a success notification or error notification will be displayed after submitting the edit form.

We overcame these challenges by upgrading the library used so we could put data- props on the associated HTML mounted to the page for any of the notifications. This way we could check the status of the notification, the text included, and giving us a way to close the notification after our test is done validating its information.

Role Management

Another part of Automatisch requiring test coverage was role management. Automatisch provides an strong Role Based Access Control (RBAC) authorization system controlling which actions a user is able to perform. There are permissions for reading, creating, updating, and deleting, for many different sets of features within Automatisch. These features RBAC controls includes creating workflows integrating various third party software together, creating new connections to third party vendors, and managing executions of the workflows.

We wrote integration tests for each of these sets of permissions, ensuring they worked as intended, and ensured the interface behaved as expected when the logged-in user did or did not have the relevant permission.

In addition to these tests on permission, we also created test coverage for the admin workflows for managing these roles and permissions. This includes the standard creating, updating, and deleting of roles. In addition, we added coverage for edge case behavior similar to the edge cases found in the administration of users.

CI Pipeline Enhancements

Automatisch serves a diverse range of users, from open source self-hosted users to enterprise clients. Being able to push updates to this diverse user base requires adequate testing covering functionality available to both sets of users. Part of this equation is testing in the CI pipeline all types of end-users, ensuring robust updates of the software. We supported Automatisch in crafting their CI pipeline with GitHub actions, giving test support for both self-hosted and enterprise features during development, ultimately building a reliable development ecosystem.

Tangible Value

Already with our test work is Automatisch reaping the benefits from these automated checks. Currently they are working on a major restructuring of their application, moving the previous GraphQL API to the current REST API. Their main lighthouse guiding them throughout this process has been their test suite, since it has given them assurance these major structural changes to their codebase are not interfering with the already existing features and workflows supported on their platform.