Category Archives: Software Engineering

Should We Rebuild or Renovate an Outdated System?

Motivation

You have an outdated system and want to rebuild or renovate it. You do not know which path is best for you.

Guidelines

Typically you have 2 main options for an outdated system.

Option 1 – Replacement: You can rebuild the whole system and migrate the existing data to the new system.

If you follow this approach then you may consider WordPress if your system is just a brochure website. WordPress is very suitable for building a brochure website.

If you system is a simple software supporting few business processes then you may consider EspoCRM. EspoCRM is very suitable for building simple business processes.

If your system is a sophisticated software supporting specific business then there may be several things you may need to consider when deciding to rebuild your system.

1. The effort to migrate the existing data to the new platform. A deep analysis of the existing data is required to create a reasonable estimate for this task.

2. The effort to integrate the new system with other systems that are integrated with the existing system. An analysis of the existing source code is required to create a reasonable estimate for this task.

3. The effort to implement all the existing business processes in the new system. A list of all business processes with their concrete steps is required to create a reasonable estimate for this task.

4. The possible mismatches of the new platform and the existing code base and data. These may require changes of business processes or removal of some existing features or considerable effort for migration.

5. Are the source code, database and files of the existing website fully available? Some websites might suffer vendor lock-in issue. It may be true that any exported data is unusable without the source code. Therefore no migration tool can help data migration due to specific data structures and constrains.

For example no one-way encrypted user passwords can be migrated to a new system. The best data that might be available for you is just Excel or CSV files. The contents of these files may be imported to the new system manually or automatically but no business logic can be transferred. In this case, rebuilding the whole system (preferably using WordPress) will require manually copying the contents to the new system and partially importing few tables (e.g. emails, first names, last names, page titles) to the new system.

Option 2 – Renovation: You may choose to renovate your system if the existing technology stack is not too outdated.

When choosing this option you may need to consider several things below.

1. The life cycle of the existing technology stack. The existing technology stack should not be deprecated and is still in active support.

2. The ecosystem of the existing technology stack. Most of 3rd party components for the existing technology stack should still be in development and support.

3. The capabilities of the existing technology stack. The existing technology stack should still support your new requirements although the solutions may not be perfect for performance or user experience.

4. The cost of the existing technology stack. You should still be able to afford the cost of the existing technology stack.

 

Should We Use SOAP or REST or gRPC or GraphQL for API Implementation?

Motivation

You want to implement an API for your client to integrate with your system.

You wonder if you should implement the API using SOAP or REST or GraphQL or gRPC.

Guidelines

Technically speaking SOAP is a protocol (developers must respect the protocol specification when working with it) while REST is an architecture style (Developers may not follow it strictly).

Most of software service providers (e.g. Salesforce, Sharepoint) provide both of them (SOAP and REST) for API clients.

The reason why a software service provider may implement both SOAP and REST is that legacy systems or frameworks (e.g. .NET, Java) often have many mature tools or libraries for consuming SOAP API easily while modern systems or frameworks (e.g. Node.js) often have many available tools/libraries for quickly consuming REST API.

If only one approach has to be chosen in order to reduce development cost then selecting SOAP or REST or GraphQL or gRPC for API implementation should depend on the API clients.

So the strategy below may help us select the most appropirate technology.

  1. Introducing your system capabilities to your client first.
  2. Then find out the main programming language that the client is using for their back-end services (e.g. Java, C#, PHP, JavaScript, Python, Ruby, Go)
  3. Find out the main API technology that the client is using for their back-end services or the API technology that they are most familiar with (e.g. SOAP, REST, GraphQL, gRPC).
  4. Then implement your API using the client’s API technology. The API can be implemented using a programing language that you are most familiar with.
  5. Then create the corresponding API developer guide and sample code for the client using their programming language.

 

How to Automate Code Signing using EV Code Signing Certificate

Motivation

Your software installer (e.g. an EXE or MSI file) is warned by Microsoft Defender. You want to remove the warning so that your users can trust your software.

Solution

You need to sign your software installer using an EV code signing certificate.

Terminologies

SSL certificate is a cryptographic certificate (technically named X.509) issued by a Certificate Authority as a file for encrypting and decrypting data between a client and server.
SSL certificate is usually valid for one domain in 1 year.

Code singing certificate is a cryptographic certificate (technically named X.509) issued by a Certificate Authority as a file for placing a digital signature on a file, program, or software.
Code signing certificate is usually valid for signing file, program or software from 1 year to 3 years.
Code signing certificate does not relate to domain.

EV code signing certificate is a code singing certificate (technically named X.509) that is issued on USB or HSM or Cloud Signing Service (not a file) by a Certificate Authority for placing a digital signature on a file, program, or software to bypass MS Defender warning.
EV code signing certificate is usually valid for signing file, program or software from 1 year to 3 years.
EV code signing certificate does not relate to domain.
When signing file, program or software using an EV code signing beside presence of USB or HSM or Cloud Signing Service an additional one-time password (OTP) is usually required for security.

EV Code Signing Certificate, USB, HSM, and Cloud Code Signing Service

The core part of a certificate is a secret string called private key.
All the certificates have this same core part. The difference is just where this string will be stored.

– This secret string can be stored in a machine/server (standard certificate) and can be exported and imported to other machines/servers, e.g. via PFX files with private keys.

– This secret string can be stored in a USB (this is a requirement for EV certificate). In this case, it can be imported to the USB but then could NOT be exported from the USB and imported to other locations.

– This secret string can be stored in a HSM (hardware security module, usually a physical card with cryptoprocessor chip or a certified virtual machine, USB can be considered a HSM too) (this is a requirement for EV certificate). In this case, it can be imported to the HSM but then could NOT be exported from the HSM and imported to other locations.

– This secret string can be stored in a Cloud Code Signing Service (this is actually a specific certified HSM virtual machine) (this is a requirement for EV certificate). In this case, it can be imported to the Cloud Code Signing Service but then could NOT be exported from the Cloud Code Signing Service and imported to other locations.

If we have 3 physical servers and we use the USB storage approach for EV Code Signing certificate then we need 3 USB sticks (duplicates) that contain the same certificate.

If we use HSM or Cloud Code Signing Service then we just need one HSM or one Cloud Code Signing Service account.

Typical steps to sign code using EV code signing certificate

– Purchase an EV code signing certificate.

– Retrieve the USB containing private key and temporary token password.

– Install software to connect to the USB, e.g. SafeNet Authentication Client, and change the temporary token password.

– Find and copy the SignTool.exe utility from Visual Studio Tools to a location.

– Enable Single Logon feature of SafeNet Authentication Client on Client Settings > Advanced so that we can pass the EV certificate password to SignTool.

– On SafeNet Authentication Client, right click the EV certificate and export the public certificate (.CER file).

– On SafeNet Authentication Client, click the EV certificate and retrieve the key container name of the EV certificate.

– Sign the software installer using the command below.

SignTool sign /f myCert.cer /csp "eToken Base Cryptographic Provider" /k "[{{TokenPasswordHere}}]=KeyContainerNameHere" myFile.msi /tr http://timestamp.digicert.com /td sha256 /fd sha256

– Verify the signing using the command below.

SignTool verify /pa myFile.msi

 

 

Microservice-Based System

Motivation:

You want to use microservice architecture for your system but you are not sure whether your design does follow microservice style.

You need to explain your microservice-based design to your client or manager.

You need to evaluate a system against microservice architecture style.

Information:

A microservice-based system contains 2 main parts: a user interface and a collection of APIs (or web services or just services).

[A diagram shall be added here.]

The user interface can be multiple web pages or single web page or mobile native application or Windows native application.

The user interface can be implemented using ASP.NET MVC or React or Angular or Android SDK or iOS SDK or even ASP.NET Web Forms or WordPress.

Each API must have its own database , must be deployed independently of other APIs, can be implemented using preferred technology stack, and should focus on a single business domain.

Each API database can be a relational database (MS SQL, Oracle, mySQL) or noSQL database (MongoDB, Cassandra).

These APIs can be implemented using .NET Framework 4.8 or .NET 6 or Spring Boot (Java) or Node.js or Golang or even WordPress.

These APIs often share an identity provider that in turn is often a specific API too.

These APIs often share several utilities (e.g. notification service, file service, payment service) that in turn are often specific APIs too.

These APIs can be grouped into API Gateways for easier authentication and authorization control. The API Gateways can be implemented using using .NET Framework 4.8 or .NET 6 or Spring Boot (Java) or Node.js or Golang or even WordPress.

These APIs can also share data with one another using a message broker (i.e. message queues). These message queues can be implemented using a relational database (MS SQL, Oracle, mySQL) or noSQL database (MongoDB) or a distributed event streaming platform (Kafka, RabbitMQ).

 

 

Why Select Microservice Architecture for Your System?

Motivation:

You need to decide whether to use microservice architecture for a new system or for modernizing a legacy system.

Suggestions:

A. Why select microservice architecture for your new system?

Reason 1: You want to have several independent development teams.

Reason 2: You want or your client wants to have a faster and modern UI (i.e. SPA) that is totally decoupled from the other parts of the system.

Reason 3: You want or your client wants that a part of your system (e.g. user store and authentication) can be shared among or reused by multiple different systems.

Reason 4: You want or your client wants your system to be able to scale out easily later.

Reason 5: You want or your client wants that some parts of your system can be revised or extended or replaced easily later.


B. Why migrate your legacy system to microservice architecture?

Reason 1: You want to have several independent development teams.

Reason 2: You want or your client wants to have a faster and modern UI (i.e. SPA) that is totally decoupled from the other parts of the system while the old technologies (like ASP.NET Web Forms or WordPress) cannot provide it.

Reason 3: You do or your client does have performance or scalability issue with the legacy system.

Reason 4: You find that is too difficult to add a new feature to the legacy system because it is too complicated.

Reason 5: You want or your client wants that some parts of your system (e.g. user store and authentication) can be shared among or reused by multiple different systems.

Reason 6: You want or your client wants your system to be able to scale out easily later.

Reason 7: You want or your client wants that a part of your system can be revised or extended or replaced easily later.


C. Here are my suggestions based on estimated development effort and a number of system users.

For a new small project (3-month effort, less than 1,000 users) I always recommend that we use any simple solution that works (e.g. WordPress, Magento, Salesforce LWC, ABP).

For a new medium or large project (2-year effort, more than 20,000 users) I recommend that we use a microservice-based and SPA solution.

For a legacy project I think that we do need a really “good” reason to migrate it to microservice architecture first.

Maybe your client wants to provide a modern SPA for their users.

Maybe the number of users grows to 1,000,000 and you do have a performance or scalability issue.

 

WordPress Lessons Learned

Lessons learned 1: Maintenance creep.

Problem:

The cost of updating the core, updating plug-ins, and updating integration code for a system built with WordPress is much higher than expected.

Context:

WordPress core, its themes and plug-ins can provide a very fast solution by integrating their features. It works best for used for achieving a general solution (for example corporate websites with predefined business processes).

However their very frequent updates are painful too.

If we do not apply the updates to the system then we may have risk of potential security issues.

Therefore the maintenance of a WordPress based system usually takes more effort than systems built using other web frameworks that do not require very frequent updates.

The problem become worse when we apply the updates to the system and the integration may be broken because of incompatibility among the plug-ins owned by different software providers and WordPress.

Solution:

If you have have opportunity of building a system for a client using WordPress and its plug-ins then you should plan a considerable maintenance budget for updating WordPress, specific plug-ins and integration code.

This lesson learned applies to any system built with a plug-in based architecture (i.e. microkernel architecture) in which the core and plug-ins owned by different software providers.

Lessons learned 2: Customization creep.

Problem:

The cost of integrating a theme with various COTS plugins and integration code for building a system using WordPress is much higher than you expected.

Context:

Customizing a WordPress solution for specific business processes takes more effort than using other general web frameworks because we do not have full control of the themes and plug-ins.

Even we can modify the themes and plug-ins or build new specific plug-ins using PHP and mySQL the solution still depends on WordPress core the architecture of which is not good for domain-specific enterprise systems which often rely much on domain-specific rules that should be isolated from specific infrastructure.

Solution:

If you have have opportunity of building a domain-specific system using WordPress and its plug-ins then you should

  • refine business requirements into appropriate user stories or use cases.
  • create an architecture first and evaluate it carefully using various scenarios, and
  • create a poof of concept.

Lessons learned 3: WordPress and Salesforce integration decision.

Motivation:

You need to evaluate options for integrating WordPress with Salesforce so that development and maintenance cost is acceptable.

Solution:

IMO an enterprise solution based on Salesforce and WordPress integration may be proposed based on a client’s needs and budget.

I define a long term project as a project many big enhancements of which will be required after the solution is deployed.

I define a short term project as a project only some small enhancements of which will be required after the solution is deployed.

I define an important project as a project the requirements of which must be satisfied exactly as the Client requires.

I define a flexible project is a project the requirements of which may be implemented not exactly as the Client requires due to constraints of tools.

If the client needs Salesforce CRM (a must requirement) and they have a big budget then I would recommend that they use Salesforce for file storage and all Salesforce tools (i.e. LWC, community portals) for implementing their own business processes for a long term and important project.

If the client needs Salesforce CRM (a must requirement) and they have medium budget then I would recommend that they use BOX or DropBox or Sharepoint for file storage and all Salesforce tools (i.e. LWC, community portals) for implementing their own business processes for a long term and important project.

If the client needs Salesforce CRM (a must requirement) and they have low budget then I would recommend that they use WordPress for file storage and Commercial off-the-shelf (COTS) plug-ins for implementing their own business processes for a short term or “flexible” project.
In this case, the development effort may be reduced much but the maintenance/enhancement effort may be increased much.

If the client does NOT really need Salesforce CRM and they have medium budget then I would recommend that they use general frameworks and tools (including WordPress and custom plug-ins) for implementing their own business processes for a long term and important project.
File storage can be local server or any service.

If the client  does NOT really need Salesforce CRM and they have low budget then I would recommend that they use WordPress for file storage and WordPress COTS plug-ins for implementing their own business processes for a short term or flexible project.

 

 

 

      How to Automate Cleaning Up a .NET Solution

      Motivation:

      You have a .NET solution with many projects.
      You need to deliver the solution to a client very frequently.
      You need to clean up all intermediate folders or files to reduce the package size and make the result look tidy.
      You want to automate this process to reduce cleaning up time.

      Solution:
      1. In Visual Studio, right click a project and select Properties.
      2. Click on Build Events.
      3. Enter the following commands to the Post-build event command line text box:
        rd /s /q $(ProjectDir)obj
        cd $(TargetDir)
        del *.config
        del *.pdb
        del *.xml

        rd /s /q $(ProjectDir)obj: Remove the obj folder in the project directory.
        cd $(TargetDir): Move to the output directory.
        del *.config: Delete all the config files in the output directory.
        del *.pdb: Delete all the pdb files in the output directory.
        del *.xml: Delete all the xml files in the output directory.

        You can modify these commands or add new commands for your specific purpose.

      4. Save the project.
      5. Repeat the process for all the projects in the solution.

      In order to delete all the obj folders recursively you can

        1. Create DeleteObjFolders.bat file inside the same folder that contains your solution file (i.e. *.sln)
        2. Paste content below to the DeleteObjFolders.bat file:
        @echo off
        @echo Deleting all OBJ folders...
        for /d /r . %%d in (obj) do @if exist "%%d" rd /s/q "%%d"
        @echo OBJ folders successfully deleted :) Press any key to close the window.
        pause > nul

        3. Execute the DeleteObjFolders.bat file.

       

      How to Quickly Estimate Project Effort using Expert Judgement?

      Motivation:

      You need to quickly estimate effort for building an enterprise system.

      Solution:
      1. Ensure that you have a requirements analysis document.
        This can be

        • list of artifacts or
        • a list of features or
        • a list of use cases or
        • a list of user stories or
        • a list of user interfaces or
        • a list of workflows with their steps.
          Note: Never conduct an estimation using expert judgement without any inputs.
      2. Break down each of item in the list into the following (but not limited to) tasks to create:
        • UI and navigation code,
        • database script or storage schema definition code,
        • API code,
        • business logic code,
        • data access code,
        • scheduler code,
        • unit tests code,
        • deployment script,
        • end-to-end test cases and test reports, and
        • user guide.
          Note: You do not need to break down a feature into tasks if you can estimate an effort to complete it with high confidence.
      3. Estimate the effort in man-days for completing each task.
        When giving an estimate for each task, think about

        • possible inputs validation and error handling aspect,
        • possible navigation framework preparation,
        • possible alternative scenarios of the feature related to the task,
        • possible encryption and decryption aspect,
        • possible attacks prevention (e.g. adding CAPTCHA, trimming special characters),
        • possible frameworks, libraries or external tools preparation, and
        • communication, meeting and status report for each task.
          Note: You should break down a task into sub-tasks if you are still not confident when giving an estimate for completing it.
      4. You do not need to perform the second and third step in a sequential order, just complete anything you can and repeat these 2 steps as many times as you like.
      5. If you cannot break down a feature into tasks or cannot give an estimated value for a task then you need to create a prototype or a proof of concept related to the feature.
      6. If you want to have a more accurate estimated value for a task then you should give 3 values for the task estimation: the best case value (optimistic value), the worse case value (pessimistic value) and the most likely value, then calculate the estimated value using the formula below.
        Estimated value = (Best case value + (4 X Most likely value) + Worse case value)/6
      7.  If you just need to give an effort guesstimate for a project containing too many requirements then you can group similar items together, then guess an effort for completing one item in a group and multiply this value with the number of items in the group, then repeat this guess and calculation for all the remaining groups.
      8. Include all communication, documentation and training effort into the guesstimate.

       

       

       

         

         

        How to Quickly Capture and Analyze Requirements for Building a Prototype or Proof of Concept?

        Motivation:

        You need to quickly capture and analyze requirements for building a prototype or proof of concept for an enterprise system.

        Suggestion:
        1. Capture any artifacts related to a business need. They may be some emails, presentation slides, a proposal document, a legacy system, some similar systems, a feature list or an initial requirement specification.
        2. Define terminologies (terms), build a background and context around the terminologies, expand the terminologies into business roles, business workflows, business components, business entities (objects), and business artifacts as much as possible.
        3. Identify the main stakeholders (roles) who will interact with the system (e.g. Guest, Admin, Developer, User, etc.) and their corresponding business problems or needs (i.e. their motivation of using the software system), and their current business workflows. Create as many scenarios (real world or domain business workflows) as possible. Create a domain model (a model of real world or domain entities, their properties and their relationships) if necessary.
        4. If there is an existing or similar system then identify the core components that each stakeholder will interact with (e.g. Accounts, Profiles, Reporting, APIs, Lessons, Videos, etc.) and their corresponding purposes.
        5. Identify the main tasks that each stakeholder will perform (e.g. Register an Account, Log in System, Create/Edit Profile, Create a Bucket, Generate an API Key, Create/Edit/Delete a Lesson, Create/Edit/Delete a Video, View Bandwidth, etc.).
        6. Identify the inputs and outputs and create mock-ups (sketches) or capture similar screens for each task. The quickest way to create a mock-up is to find similar existing screens in your system or external systems, and modify them. You can also search for templates of similar features in the Internet, and modify them. You can also brainstorm a  new one if you are good at creativity and imagining.
        7. Build or draw a prototype demonstrating 2 or 3 critical end-to-end workflows (An end-to-end workflow is a sequence of tasks that solves a real world problem completely). Contrast a current business workflows with the end-to-end workflows demonstrated by the prototype. The quickest way to create a prototype is to find similar existing workflow of of your system or external systems, and modify them. You can also repurpose a business workflow for a prototype. You can also brainstorm a  new one if you are good at creativity and imagining.
        8. Implement a proof of concept related to the 2 or 3 critical end-to-end workflows.
        9. For a complicated system, you may need to implement a proof of concept, build a prototype, and create a work break down structure or product backlog in parallel to ensure that the requirements are technologically and economically feasible.

         

         

        How to Quickly and Confidently Read Code?

        Motivation:

        You need to read existing source code to add your feature or fix a bug or to replicate its functions in another language or system.

        Solution:
        1. Try building and running the code without any modification.
        2. Try building and running the code with your inputs.
        3. Try modifying or adding simple UI elements without touching the business logic or data layer.
        4. Try writing a unit test for a function.
        5. Try changing a variable name and make sure that the change does not break anything.
        6. Try changing a package or module name and make sure that that the change does not break anything.
        7. Try breaking a long method to smaller ones.
        8. Try breaking a large object or file to smaller ones.
        9. Try replacing an algorithm with with a better one.
        10. Try replacing an old component with a newer or better one.
        11. Try recreating the code structure from the separate components. If you cannot do this then try to understand the architecture of the existing source code.
        12. If you get an error when making the changes above then try showing the error on client side.
        13. If you cannot show the error on client side then try identifying the logic flow of the code related to the error, debug information, and learn about new terminologies or concepts in the code.

        Example:

        const maskPhone = (val) => {
            const x = val.replace(/\D+/g, '').match(/(\d{0,3})(\d{0,3})(\d{0,4})/);
            return !x[2] ? x[1] : `(${x[1]}) ${x[2]}` + (x[3] ? `-${x[3]}` : ``);
        };
        export { maskPhone }

        If you are not sure about what the JavaScript code above does then try to learn

        Another example:

        <div class="container">
            <iframe class="responsive-iframe" src="https://www.youtube.com/embed/u9Dg-g7t2l4"></iframe>
        </div>
        .container {  
            position: relative;  
            overflow: hidden;  
            width: 100%;  
            padding-top: 56.25%; /* 16:9 Aspect Ratio (divide 9 by 16 = 0.5625) */
        }
        /* Style the iframe to fit in the container div with full height and width */
        .responsive-iframe {  
            position: absolute;  
            top: 0;  
            left: 0;  
            bottom: 0;  
            right: 0;  
            width: 100%;  
            height: 100%;
        }
        

        If you are not sure about what the HTML and CSS code above  do then try to learn

        • how many ways an element can be placed in a web page (a document),
        • what is the position relationship between an element and its parent,
        • what is the position relationship between an element and its siblings,
        • what is the position relationship between an element and the document root, and
        • what position: relative , position: absolute , overflow: hidden do.