All posts by mahieddine.chir

I just want to put another brick on the wall, hoping that I could help people with my poor, but I hope useful, experience about computer science, data analysis ...

A Custom @AutoConfigure to provide a bean (Spring BeanPostProcessor likewise) for Unit testing dependent classes

This article is to show u how to implement your own bean factory on a unit tests (BeanPostProcess like wise) using @AutoConfigure features of _Spring (Boot)_.

Context

I had this need when I tried to implement a Unit test for a Custom repository class (using Spring Boot) that implements some logic, and is dependent on _EntityManager_.


@Repository
public class CustomRepository {
    @EntityManager
    entityManager;

    public AnyOutput any(AnyInput input) {
      // logic implementation
    }
}

I could, of course, _Mock_ the _EntityManager_ … but I needed something that validates the queries that were built dynamically from input parameters.

Why I did not use @DataJpaTest

Simply because this latter is made for Integration tests, launching a micro container and trying to deploy and validate all repository classes I have in my project! My need was a Unit test for my `CustomRepository`, not else.

Implementation details

  1. The in memory engine bean factory:
  2. For that, I’ve chosen to use HSQLDB in memory database as a persistent engine providing the following bean factory

    
    @Configuration
    public class HSQLDBEntityManagerConfiguration {
    
      @ConditionalOnMissingBean
      @Bean
      public EntityManager entityManager(@Autowired @Qualifier("test-entities") Collection<Class<?>> entities) {
        org.hibernate.cfg.Configuration configuration = new org.hibernate.cfg.Configuration();
        entities.forEach(aClass -> configuration.addAnnotatedClass(aClass));
        configuration.setProperty("hibernate.connection.driver_class", org.hsqldb.jdbcDriver.class.getName())
          .setProperty("hibernate.connection.url", "jdbc:hsqldb:mem:testdb")
          .setProperty("hibernate.connection.username", "sa")
          .setProperty("hibernate.connection.password", "")
          .setProperty("hibernate.dialect", org.hibernate.dialect.HSQLDialect.class.getName())
          .setProperty("hibernate.hbm2ddl.auto", "update")
          .setProperty("hibernate.show_sql", "true")
          .configure();
    
        SessionFactory sessionFactory = configuration.buildSessionFactory(
          new StandardServiceRegistryBuilder()
            .applySettings(configuration.getProperties())
            .build()
        );
        Session session = sessionFactory.openSession();
        Runtime.getRuntime().addShutdownHook(new Thread(sessionFactory::close));
        EntityManagerFactory entityManagerFactory = session.getEntityManagerFactory();
        EntityManager entityManager = entityManagerFactory.createEntityManager();
    
        return (EntityManager) Proxy.newProxyInstance(this.getClass().getClassLoader(), new Class[]{EntityManager.class}, (proxy, method, args) -> {
    
          if (! method.getName().equalsIgnoreCase("close")) {
            entityManager.getTransaction().begin();
          }
          Object invoke = method.invoke(entityManager, args);
          if (! method.getName().equalsIgnoreCase("close")) {
            entityManager.getTransaction().commit();
          }
    
          return invoke;
        });
      }
    }
    
    

    The `@Qualifier(“test-entities”) Collection<Class> entities` here are needed for an annotated based Hibernate session factory configuration. This is expected to be provided in the Spring registry at runtime as we will see later.

    This is of course an example detail, … your bean factory could me much simpler!

  3. The Spring-Boot likewise auto-configuration
  4. 
        @Retention(RetentionPolicy.RUNTIME)
        @Target(ElementType.TYPE)
        @ImportAutoConfiguration(HSQLDBEntityManagerConfiguration.class)
        public @interface AutoConfigureEntityManagerHSQLDB {
        }
    
    
  5. Usage example of the above in a Unit test
  6. 
    @RunWith(SpringRunner.class)
    @AutoConfigureEntityManagerHSQLDB // this what makes the in-memory entity manager available for DI
    @Import(DuplicateRepositoryTest.Config.class)
    public class CustomRepositoryTest {
    
        @Autowired
        CustomRepository duplicateRepository;
    
        @Autowired
        EntityManager entityManager;
    
        @Before
        public void init() {
            // test context and data setup 
            SubEntity subEntity = new SubEntity();
            subEntity.setName("sub_entity");
    
            TestEntity entity = new TestEntity();
            entity.setName("test_entity");
            entity.setSubEntity(subEntity);
    
            entityManager.persist(entity);
        }
    
        @Test
        public void onDuplicateShouldReturnTrue() {        
            // test logic implementation and assertions
        }
    
        @Configuration
        @Import(CustomRepository.class)
        static class Config {
    
            // these are the entities configured for the annotation based session factory initialisation
            // TestEntity and SubEntity are marked with JPA annotations (@Entity, ... etc)
            @Bean
            @Qualifier("test-entities")
            Collection<Class<?>> getEntities() {
                return Arrays.asList(TestEntity.class, SubEntity.class);
            }
        }
    }
    
    
Advertisements

A Keycloak Spring Security (xml) library for securing web-applications (not using introspection) but signature validation

I recently worked on keycloak integration in a set of Restfull API’s for OAuth2 token based authentication … and I wanted to share with you a lightweight spring-security-keycloak library I developed for managing this.

Important to note that your web application need not be necessarily developed in spring(-web) as spring security is highly pluggable in any Servlet base web application.

Security configuration

The security definition (routes security declarations) lies in src/main/resources/token-security.xml.
I still prefer using spring security namespace config’s as I prefer keeping these cross concerns apart from code.

Routes config

The config defines three routes securities

  • /public/** non secured resources (static, helpers, … that are
    accessible from a non secured web application area)
  • /private/** secured, accessible by Client Credentials OAuth2
    grant_type tokens (generally described as Service Accounts). Remember that
    these tokens are issued using only applications clientId and clientSecret.These _logged_ entities are expected
    to contain _APPLICATION_ Role. This role is extracted from the `realm_access.roles` (keycloak) token claim
    (remember that keycloak generates JWT tokens format)

    The /private/** are generally reserved when systems want to access their resources
    (API to API calls for example)

  • /** secured accessible by users as their resources (standard Restfull resources).
    Such endpoints are accessible by tokens obtained from an OAuth2 or an OpenId Connect server
    using standard web flows (Authorization Code or Implicit flow) or (not recommanded) Password credentials
    flow
    These logged user in this scope are expected
    to contain _USER_ Role. Also extracted from the realm_access.roles (keycloak) token claim

Using the lib in your web module

  1. Add the project as a jar dependency
  2. configure the servlet in your `web.xml` as follows
  3. <?xml version="1.0" encoding="UTF-8"?>
       <web-app>
        <!-- Spring security config -->
       	<filter>
       		<filter-name>springSecurityFilterChain</filter-name>
       		<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
       	</filter>
       	<filter-mapping>
       		<filter-name>springSecurityFilterChain</filter-name>
       		<url-pattern>/*</url-pattern>
       	</filter-mapping>   
       	<listener>
       		<listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
       	</listener>
       	<context-param>
       		<param-name>contextConfigLocation</param-name>
       		<param-value>
       	        classpath:/token-security.xml
       	    </param-value>
       	</context-param>
        </web-app>
    
  4. Do not forget to specify these two config parameters when starting you application

Through environment variables for example (generally preferred in cloud environments) or through standard
application.properties|yml spring config files.

Important to note that in this library, we are not using the keycloak introspection endpoint as
calling a remote http server on every received token may be rapidly an issue, especially in full API micro service
applications … loosing the benefit of the JWT format auto-signed tokens.

Sign keys retrieved from the keycloak and generated public keys objects are cached.

  • Project github link here https://github.com/mahieddine-ichir/spring-security-keycloak
  • Starting a (basic) Jetty Server from Java

    for information, I used this in a Spring Boot Application for RPC communication between Kafka instances for remote Ktable data retrieval based on Metadata

    package net.myjettyserver;
    
    import org.eclipse.jetty.server.Request;
    import org.eclipse.jetty.server.Server;
    import org.eclipse.jetty.server.handler.AbstractHandler;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.beans.factory.annotation.Value;
    import org.springframework.context.annotation.Profile;
    import org.springframework.http.HttpStatus;
    import org.springframework.stereotype.Component;
    import org.springframework.util.StringUtils;
    
    import javax.annotation.PostConstruct;
    import javax.annotation.PreDestroy;
    import javax.servlet.ServletException;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;
    import java.io.IOException;
    
    /**
     * Created by michir on 14/05/2018.
     */
    @Component
    @Profile("rpc")
    public class RpcEndpoint {
    
        private static final String context = '/myapi'; // can be passed from command line
    
        // RPC Endpoint (listening host and port) in the format localhost:3000
        // can be passed from command line "--rpcEndpoint=localhost:3000"
        @Value("${rpcEndpoint}")
        private String rpcEndpoint;
    
        private Server server;
    
        @PostConstruct
        public void start() {
            try {
    
                this.server = new Server(Integer.parseInt(rpcEndpoint.split(":")[1]));
                server.setHandler(new AbstractHandler() {
    
                    @Override
                    public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {                    
                        String uri = request.getRequestURI(); // in the format '/myapi/'
    
                        // process if "GET" method
                        if (request.getMethod().equalsIgnoreCase("get") &amp;&amp; uri.startsWith(context)) {
                            String id = uri.substring(context.length()+1);
                            if (StringUtils.isEmpty(id)) {
                                response.setStatus(HttpStatus.BAD_REQUEST.value());
                            } else {
                                // do something with the id
                            }
                        }
                    }
                });
    
                server.start();
    
                Runtime.getRuntime().addShutdownHook(new Thread(() -&gt; {
                    try {
                        server.stop();
                    } catch (Exception e) {
                        e.printStackTrace(); // can be ignored or WARN level logged
                    }
                }));
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
    
        @PreDestroy
        public void stop() {
            try {
                if (!server.isStopped()) {
                    this.server.stop();
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
    

    Go-lang Rest API on MongoDB

    I am getting more deeply in Golang and I am really appreciating it’s simplicity compared to Scala. It reminds me NodeJS / ExpressJS in some of its aspects when writing down APIs.

    And of course, if you have practiced a little GO language, you find lots of similarities with the C language (Go-ogle comes from C for sure ;-)).

    It is worth trying to learn it … all benchmarks are plebisciting it!

    In this public github repository

    https://github.com/mahieddine-ichir/golang-rest-mongo

    you can find an example Rest API application with storage on Mongo DB developped in GO, exposing classical endpoints (GET, POST, DELETE, OPTIONS (for CORS)).

    You can easily adapt it to your needs and model.

    Coming next “go-lang Rest API on Kafka Broker”

    AWS : Adding swap to a running instance

    This is nothing but a summary of the following guide on how to add a swap volume to a running instance on AWS :

    By default instances started on AWS do not have allocated swap memory … but u should ALWAYS add a swap mem to your running instance to avoid lack of memory overhead!

    1. Check if you have any swap on your instance
      swapon -s
    2. If none, create a volume and attach it to the instance (you will have to assign it a letter in the format ‘sdf’, ‘sdg’, … etc. The added volume will be mounted on ‘/dev/xvdf’, ‘/dev/xvdg’, … etc. Here the example is run on ‘sdg’ mounted on ‘/dev/xvdg’.
    3. Execute
      lsblk

      to check your device (example ‘/dev/xvdg’)

    4. format the device for swap using
      sudo mkswap /dev/xvdg
    5. then execute

      sudo swapon /dev/xvdg
      
    6. Re-execute

      swapon -s

      to see your swap created!

    7. Modify your ‘/etc/fstab’ and add the following line

      /dev/xdvg       none    swap    sw      0       0
      

      to make the mount of the device persistent.

    Maven release plugin configuration on Jenkins

    Maven configuration file settings.xml on the Jenkins server

    Insert the following profile in the settings.xml that is on your Jenkins server:

    
    <profile>
      <id>gitlab</id>
      <properties>
          <project.scm.id>gitlab</project.scm.id>
      </properties>
    </profile>  
    
    

    And insert the following server configuration in the same file

    
    <server>
       <id>gitlab</id>
       <username>your-gitlab-username<username>
       <password>your-gitlab-password<password>
    </server>
    
    

    Job configuration on the Jenkins server

    1. Install the Maven Release Plug-in Plug-in
    2. Configure the build job to add the profile to the maven command line
      -Pgitlab

    You can now enjoy performing releases from your Jenkins server.

    JBoss 4.2.3.GA – Administration guide best practices

    The JBoss AS 4.2.3 ships with the following directory structure for a given instance:

    conf/     contains JBoss bootstrap configuration files
    lib/      contains static libraries that are loaded on startup
    deploy/   contains deployable resources (applications, xml files, jar files, ...)
    ...
    

    When going to production, people just ships the JBoss AS as it is and deployments are operation on this directory structure directly. I think this is a bad practice and I personnaly strongly advice to separate the concerns and isolate application specific resources into separate and external folders:

    • these folders contain instance bootstrap resources and modifying or manipulating these resources can be tricky
    • Separating JBoss AS resources from application specific resources will make any deployment automation more easier
    • Isolating JBoss AS resources will make Docker-ization of a JBoss container much more easier too!

    Hopefully this has been made possible by JBoss!

    The deployment folder “./deploy”

    The JBoss AS “deploy” folder of an instance ships with a certain number of applications (a EJB Deployer, a WEB deployer – namely Tomcat, JMX console …).
    This folder is scanned on startup and is configured in the conf/jboss-service.xml file. You can add any number of external folders to the scanning component for application deployment on startup as well as for hot deployment. Locate the section below within the org.jboss.deployment.scanner.URLDeploymentScanner

        <attribute name="URLs">
             deploy/
        </attribute>
    

    and just add you (external to jboss) applications specific deployment folder as follows:

        <attribute name="URLs">
             deploy/,
             /path_to_deployment_folder
        </attribute>
    

    In Docker, this folder can be mapped as a Volume.

    The configuration “./conf”

    The ./conf folder, in addition to containing JBoss bootstrap configuration files can contain your application specific resources, these resources are added to the applications resources classpath by JBoss and can be loaded using, for example, using

    MyClass.getResourceAsStream("/myresource");
    

    You can define, through the command line an external folder to the loaders classpath as follows:

    bin/run.sh --classpath=/path_to_resources_folder
    

    You can put in this folder all the ressources (xml files, properties, … etc) the application will access through MyClass.getResourceAsStream("/myresource");.

    In Docker, this folder can be mapped as a Volume.

    The static libraries folder

    The ./lib folder contains static libraries (loaded on startup), these libraries are necessary for JBoss default applications deployment and startup.
    For your application libraries, you can either package them within the artifact (ear or war) – I strongly advice this approach – or you can add them as static library and be loaded on JBoss startup (this is mandatory if you want to deploy a .jar artifact for example as this latter cannot contain third party libraries).

    You can define a separate and external libraries folder for you application third party libraries though command line as follows

    bin/run.sh --library=path_to_libraries_folder
    

    In Docker, this folder can be mapped as a Volume.

    JBoss 4.2.3.GA command line options

    Here is the list of JBoss 4.2.3.GA launch options from the MainClass of the bin/run.jar

    -h, --help                    Show this help message
    -V, --version                 Show version information
    --                            Stop processing options
    -D<name>[=<value>]            Set a system property
    -d, --bootdir=<dir>           Set the boot patch directory; Must be absolute or url
    -p, --patchdir=<dir>          Set the patch directory; Must be absolute or url
    -n, --netboot=<url>           Boot from net with the given url as base
    -c, --configuration=<name>    Set the server configuration name
    -B, --bootlib=<filename>      Add an extra library to the front bootclasspath
    -L, --library=<filename>      Add an extra library to the loaders classpath
    -C, --classpath=<url>         Add an extra url to the loaders classpath
    -P, --properties=<url>        Load system properties from the given url
    -b, --host=<host or ip>       Bind address for all JBoss services
    -g, --partition=<name>        HA Partition name (default=DefaultDomain)
    -u, --udp=<ip>                UDP multicast address
    -l, --log=<log4j|jdk>         Specify the logger plugin type