#Clojure , #MongoDB and removing a set of items

When using congomongo library to access MongoDB in Clojure, it is sometimes required to remove a collection of items. As the example on GitHub page shows only how to remove one item, I have pasted the code here.

Setup test data

To setup test data, we would use 100 items in :points collection.

(use 'somnium.congomongo)
 
(mass-insert! :points
  (for [x (range 0 10) y (range 0 10)] {:x x :y y}))
 
(count (fetch :points))
;=> 100

Please note, that I am not writing any code specific to establishing a connection here. For more information on using congomongo library, please visit congomongo github site.

The obvious way

The most obvious way to do so, would be to pass a list objects to destroy! function. Sadly, it won't work:

user=> (destroy! :points (fetch :points))
;=> ClassCastException clojure.lang.LazySeq cannot be 
;=> cast to com.mongodb.DBObject  
;=> somnium.congomongo/destroy! (congomongo.clj:419)

Using for loop

We can also invoke destroy! a hundred times:

(count (fetch :points))
;=> 100
 
(for [p (fetch :points)]
  (destroy! :points p))
 
(count (fetch :points))
;=> 0

It will work, but it is not optimal.

Using where clause

The best way to remove a collection of items in MongoDB using congomongo, it would be to use where clause for a destroy! function:

(count (fetch :points))
;=> 100
 
(destroy! :points {:x {:$gt 4}})
 
(count (fetch :points))
;=> 50

It is also possible to refer MongoDB items by their ids in where clause:

(count (fetch :points))
;=> 100
 
(destroy! :points {:_id {:$in (map :_id (fetch :points))}})
 
(count (fetch :points))
;=> 0

Hacking Activiti BPM engine: how to use custom MyBatis queries

The issue

One big change that Activiti has introduced over jBPMv4 was to forego Hibernate ORM and use MyBatis. MyBatis is simpler and cleaner to use, but - it does not provide simple interface to perform queries that are not defined in annotations or XML files.

And even though Activiti provides great query interface, so you can do things like this:

List<Task> tasks = getProcessEngine().getTaskService().createTaskQuery()  
    .taskName(taskName)  
    .executionId(taskExecutionId)  
    .taskAssignee(user.getLogin())  
    .listPage(0, 1);

with relative ease, there are still some limitations to the API provided by Activiti Service classes (e.g. you cannot add IN clause for task assignees).

Possible solutions

One thing that you can do, is to just skip Activiti interface, and access database directly. Surely it will work, but such approach has many downsides - you will be fetching information about Activiti processes in two different ways, you will have to map data to Activiti objects yourself, and so on.

What we have managed to do for Aperte Workflow <-> Activiti integration (in version 2.0, where API is a bit more demanding), is indeed a hack. It is not as elegant as constructing query for jBPM, but still is quite simple and manageable. It propably can be done with Spring in a similiar fashion, or with SelectBuilder - but the principle is similiar.

The hack

1. Enhance Activiti's MyBatis mapping file - by adding our own version in a different package.

The enhanced file looks almost like the original - with one exception:

<?xml version="1.0" encoding="UTF-8"?>  
    
 <!DOCTYPE configuration PUBLIC "-//mybatis.org//DTD Config 3.0//EN" 
  "http://mybatis.org/dtd/mybatis-3-config.dtd">  
    
 <configuration>  
 <settings>  
 <setting name="lazyLoadingEnabled" value="false" />  
 </settings>  
 <mappers>  
     <mapper resource="org/activiti/db/mapping/entity/Attachment.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Comment.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Deployment.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Execution.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Group.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/HistoricActivityInstance.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/HistoricDetail.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/HistoricProcessInstance.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/HistoricTaskInstance.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/IdentityInfo.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/IdentityLink.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Job.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Membership.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/ProcessDefinition.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Property.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Resource.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/TableData.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/Task.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/User.xml" />  
     <mapper resource="org/activiti/db/mapping/entity/VariableInstance.xml" />  
     <mapper resource="org/aperteworkflow/ext/activiti/mybatis/Task-enhanced.xml" />  
 </mappers>  
 </configuration>

And the addition is:

<mapper resource="org/aperteworkflow/ext/activiti/mybatis/Task-enhanced.xml" />

2. Create enhanced mapper resource

This mapper resource add additional query conditions to the initial task query. The beginning of it is the same as original, but at the end we have added support for in clauses on several fields:

<!-- new conditions -->  
                 <if test="owners != null && owners.size() > 0">  
                     and T.OWNER_ IN  
                     <foreach item="owner" index="index" collection="owners"  
                              open="(" separator="," close=")">  
                         #{owner}  
                     </foreach>  
                 </if>  
                 <if test="notOwners != null && notOwners.size() > 0">  
                     and T.OWNER_ NOT IN  
                     <foreach item="owner" index="index" collection="notOwners"  
                              open="(" separator="," close=")">  
                         #{owner}  
                     </foreach>  
                 </if>  
                 <if test="groups != null && groups.size() > 0">  
                     and T.ASSIGNEE_ is null  
                     and I.TYPE_ = 'candidate'  
                     and I.GROUP_ID_ IN  
                     <foreach item="group" index="index" collection="groups"  
                              open="(" separator="," close=")">  
                         #{group}  
                     </foreach>  
                 </if>  
                 <if test="taskNames != null && taskNames.size() > 0">  
                     and T.NAME_ IN  
                     <foreach item="name" index="index" collection="taskNames"  
                              open="(" separator="," close=")">  
                         #{name}  
                     </foreach>  
                 </if>  
             </foreach>  
         </where>  
     </sql>  

The entire mapper file can be downloaded here from Aperte Workflow's github repository - core/activiti-context/src/main/resources/org/aperteworkflow/ext/activiti/mybatis/Task-enhanced.xml.

3. Introduce new configuration to MyBatis

As MyBatis runs inside of Activiti internals, we have to access and alter Activiti configuration mechanisms. To do that, we simply override one of the methods with our copy (which points to a new mapping file):

 public class CustomStandaloneProcessEngineConfiguration 
           extends StandaloneProcessEngineConfiguration {  
    
         @Override  
         protected void initSqlSessionFactory() {  
             if (sqlSessionFactory == null) {  
                 InputStream inputStream = null;  
                 try {  
                     inputStream = ReflectUtil.getResourceAsStream(
                       "org/aperteworkflow/ext/activiti/mybatis/mappings-enhanced.xml");  
    
                     // update the jdbc parameters to the configured ones...  
                     Environment environment = new Environment("default", transactionFactory, 
                                                               dataSource);  
                     Reader reader = new InputStreamReader(inputStream);  
                     XMLConfigBuilder parser = new XMLConfigBuilder(reader);  
                     Configuration configuration = parser.getConfiguration();  
                     configuration.setEnvironment(environment);  
                     configuration.getTypeHandlerRegistry().register(VariableType.class, 
                             JdbcType.VARCHAR,  
                             new IbatisVariableTypeHandler());  
                     configuration = parser.parse();  
    
                     sqlSessionFactory = new DefaultSqlSessionFactory(configuration);  
    
                 } catch (Exception e) {  
                     throw new ActivitiException(
                      "Error while building ibatis SqlSessionFactory: " + e.getMessage(), e);  
                 } finally {  
                     IoUtil.closeSilently(inputStream);  
                 }  
             }  
         }  
     }  

You can of course do lots of amazing customizations here. And use this new class for Activiti BPM engine initialization:

CustomStandaloneProcessEngineConfiguration customStandaloneProcessEngineConfiguration =   
         new CustomStandaloneProcessEngineConfiguration();  
         customStandaloneProcessEngineConfiguration  
         .setDatabaseSchemaUpdate(ProcessEngineConfiguration.DB_SCHEMA_UPDATE_TRUE)  
         .setDataSource(getDataSourceWrapper(sess))  
         .setHistory(ProcessEngineConfiguration.HISTORY_FULL)  
         .setTransactionsExternallyManaged(true); 

4. Enhance query object

Activiti uses wonderful, clean and simple query objects, that provide structure to MyBatis query parameters. To introduce new parameters, all we have to do, is to enhance TaskQueryImpl class with our own:

 package org.aperteworkflow.ext.activiti.mybatis;  

 import org.activiti.engine.impl.TaskQueryImpl;  
 import java.util.HashSet;  
 import java.util.Set;  
 /** 
  * @author tlipski@bluesoft.net.pl 
  */  
 public class TaskQueryImplEnhanced extends TaskQueryImpl {  
     private Set<String> creators = new HashSet<String>();  
     private Set<String> owners = new HashSet<String>();  
     private Set<String> groups = new HashSet<String>();  
     private Set<String> notOwners = new HashSet<String>();  
     private Set<String> taskNames = new HashSet<String>();  
    
     public Set<String> getGroups() {  
         return groups;  
     }  
    
     public Set<String> getNotOwners() {  
         return notOwners;  
     }  
    
     public Set<String> getOwners() {  
         return owners;  
     }  
    
     public Set<String> getCreators() {  
         return owners;  
     }  
    
     public Set<String> getTaskNames() {  
         return taskNames;  
     }  
    
     public TaskQueryImplEnhanced addTaskName(String name) {  
         taskNames.add(name);  
         return this;  
     }  
     public TaskQueryImplEnhanced addOwner(String login) {  
         owners.add(login);  
         return this;  
     }  
    
     public TaskQueryImplEnhanced addGroup(String name) {  
         groups.add(name);  
         return this;  
     }  
    
     public TaskQueryImplEnhanced addNotOwner(String login) {  
         notOwners.add(login);  
         return this;  
     }  
    
     public TaskQueryImplEnhanced addCreator(String login) {  
         creators.add(login);  
         return this;  
     }  
 }  

5. Invoke new query

And finally, we can invoke our new query with multiple assignees and other custom where clauses:

final TaskQueryImplEnhanced q = new TaskQueryImplEnhanced();  
 for (UserData u : filter.getOwners()) {  
     q.addOwner(u.getLogin());  
 }  
 for (UserData u : filter.getCreators()) {  
     q.addCreator(u.getLogin());  
 }  
 for (UserData u : filter.getNotOwners()) {  
     q.addNotOwner(u.getLogin());  
 }  
 for (String qn : filter.getQueues()) {  
     q.addGroup(qn);  
 }  
    
 ActivitiContextFactoryImpl.CustomStandaloneProcessEngineConfiguration 
   processEngineConfiguration = getProcessEngineConfiguration();  
 CommandExecutor commandExecutorTxRequired = processEngineConfiguration
   .getCommandExecutorTxRequired();  
 List<Task> tasks = commandExecutorTxRequired.execute(
   new Command<List<Task>>() {  
     @Override  
     public List<Task> execute(CommandContext commandContext) {  
        return commandContext.getDbSqlSession()
         .selectList("selectTaskByQueryCriteria_Enhanced", q);  
     }  
 });  

Please note, that we are also exposing CommandExecutor object instance to access DbSqlSession.

Summary

The technique presented here would be unnecessary or much simpler if Activiti would provide external means to configure MyBatis. Maybe that is a thing, that will be available in the future versions of Activiti.

Still, even at this moment, it is fairly easy to enhance/alter Activiti's behaviour. I haven't seen any final classes (a common sight in Hibernate - e.g. org.hibernate.impl.SessionImpl) and the internals of Activiti are quite simple to understand.

Why Vaadin is different - and how can it make your web application great

on the example of Aperte Workflow
Vaadin
Vaadin is an Open Source widget-oriented, Java Rich Internet Application web framework, which allows programmers to develop browser-based user interface entirely in Java - server-side.
That's it. This is what makes Vaadin different and this is what is game-changing in comparison to almost any other popular web framework.

100% JVM

Because everything is written in Java and runs server-side, you can do things, that were almost impossible or very painful for web frameworks before:

• Have one, coherent application, with its logic seperated according to your needs - not to what the framework designers had in mind
• Apply Java programming patterns (compile-time polymorphism, inheritance, etc.) to all of your code - and build your application from coherent modules using these patterns.
• Write app in a scripting language of your choice - in Scala, Clojure, Common Lisp, Groovy - anything that runs on JVM as a scripting language
• Debug user interface logic using standard tools - not just in a sandbox mode
• Introduce plugins to your application using OSGi framework
• Develop rapidly, by reloading your code in mere seconds - e.g. using JRebel agent or even OSGi framework

Vaadin still allows programmers to access the browser side - but in a controlled way, using Add-ons - Vaadin Add-on Directory provides ecosystem for those. And for most applications - you can easily build them using just the standard ones, provided by Vaadin out-of-the-box.

...and more

JVM itself would be enough, but Vaadin is not only great in the idea itself, but also in implementation. The applications have eye-friendly look out of the box, you can also apply custom CSS and themes easily.
The UI is well-optimized even in comparison to client-side only frameworks - and it works on all modern browsers out of the box (IE6/7 - with some downsides too!).
There is even mobile device support with Vaadin TouchKit, Vaadin also supports JSR168/286 portals (e.g. Liferay Portal) and even Google App Engine.

Vaadin & Aperte Workflow

Aperte Workflow is an Open Source BPMS - the application that eventually is built during the introduction on BPM-class system to an enterprise. Aperte Workflow is designed with strong emphasis on modularity, re-usabiliti and extensibility of solutions created.
User interface in Aperte Workflow is built on Vaadin. The approach for BPM task UI concept in Aperte Workflow is that for each human task, user interface is built from reusable blocks or components (in Aperte Workflow terminology: widgets). Such components can be process-aware (e.g. display process instance data) or just organize other components.

With Vaadin, creating these components is a blast, even for someone new to the concept. You just code your component in Java - implementing some interfaces or extending ready-to-use classes. Vaadin widgets are there - so, for example - you don't have to worry on how to implement your own or incorporate existing suggesting combo box or grid - they are already here, well designed, optimized to work together with other combo boxes.
After coding - you just deploy/update them to a plugin mechanism. Being written entirely in Java, we can manipulate them in our plugins very easily. We don't have to build a set of JavaScripts to provide RIA experience to the users - but neither we have to refresh web page after every user action.

So the code->deploy->test->code->deploy->... cycle is very short - and still we write our application in plain old Java. I wouldn't even attempt to try to implement such thing with pure GWT or JSF.
Plugin mechanism provides integration with an Aperte Workflow web-based modeler - which is based on Signavio core, but more complex elements specific for Aperte Workflow were written in Vaadin too.

Use of Vaadin in modeler allowed us also to engineer preview mechanics to some components (more to come) - for example, Advanced Process Data widget, which provides generic-form feature (it's just one plugin, not an obligation or part of the Aperte Workflow core - you can use, you can replace it, or you can use other implementation, or just go with your own plugins) generates preview in real-time. And the preview is the same code and configuration that will be run in the real user interface for a real BPM task.
What else Vaadin has allowed us to do with relative ease, is that Advanced Process Data Widget provides JSR223 scripting support - for validation and manipulation of the configured form.

Summary

In Aperte Workflow, we have chosen Vaadin as our UI framework, not only to save ourselves from spending too much time coding the user interface. We have chosen it mainly because with Vaadin, it is possible to create much more modular RIA web application than with any other framework we have seen.
Will Aperte Workflow be possible without Vaadin. Sure - but with much bigger effort and budget. And the result would be underwhelming without Vaadin - or we would have created our own, imperfect implementation of Vaadin.
More information:

• Vaadin website
• Book of Vaadin (there is even an ebook version for Kindle, iPad, Android, etc.)
• Vaadin widgets gallery
• Vaadin Add-ons directory
• Aperte Workflow

Aperte Workflow - why another BPMS?

What is Aperte Workflow?

Aperte Workflow is a BPMS (Business Process Management Suite) designed and developed by BlueSoft. The project has started in 2011 and is currently approaching version 2.0.

BlueSoft is an Independent Software Vendor specializing in custom, tailor-made IT solutions for the enterprise customers - based either on Open Source or proprietary software.

The company itself comes with over 9 years of experience in developing such solutions, which gives a certain perspective on how things should be done. But still, young enough to be able to innovate effectively.

Why build another Open Source BPM solution?

That is a good question, and we ask it ourselves every time we forego existing, available technology and start creating something new. 

With Open Source BPMs and Aperte Workflow, the decision came from a simple observation: that every time you need to deploy a BPM-based solution (be it Open Source or commercial software) in a real application, which provides real processes, used by real users, you need to build an application around it.

You have to provide at least:

• Integration with Customer's authentication/authorization mechanisms.
• Custom User Interface. Forms with text, number and date field are great, but usually the business requirements are much more complicated. 
• Integration on the browser-level with Customer's intranet portal (desktop based BPMS in my opinion are a total misunderstanding) - or some kind of base for BPM application.
• Enterprise integration with the Customer's other IT systems (CRM, ERP, ...).
• Flexible data model

And yes, it is possible to build such solution yourself. Integrate existing technologies into one solution, learn how to use them, find bugs in your code and in the code of the technologies you are using - and then, finally you can start developing your processes. 

This is exactly what Aperte Workflow provides - a ready-to-use, well-tested and mature product, which provides a platform or a placeholder for your processes and custom mechanisms. Done right, with reusability, scalability and performance in mind.

We do not aim to it be a complete solution for your business process (although it is very easy to build one with Aperte Workflow!) - we understand, that every process in every company can be a little different. But, there are some things that have to be done for every BPM deployment, so why not do them once and right?

For us, Aperte Workflow is different from the other solutions, as it approaches problems on a different level. We do not try to provide an answer to every problem that rises during BPM deployment in an organization - but rather to provide means, that the solution will be worked out easily and will be coherent with other parts of the deployment and will be re-usable in other areas.

What Aperte Workflow provides?

Aperte Workflow provides common things that usually have to be built when deploying BPM in an organization. But, since this is a standalone product, even more features, that wouldn't be usually implemented in a custom BPM deployment are provided.

Portal Integration

Aperte Workflow integrates with JSR168/286 compliant portals (e.g. Liferay). By doing so, you do not only have an awesome integration with existing portal or enterprise IT infrastructure (e.g. Active Directory, NTLM, ...), but you also have a place to introduce and integrate other applications.

Using JSR168/286 allows you to standarize and unify all your browser-based applications - into a one, coherent intranet portal. 

Custom User Interfaces

With Aperte Workflow, user interface for each human task (on the task list or in details) is built from re-usable components. These components, provided as plugins, can provide:

• all-round, generic features, such as simple input form or a tabsheet organizing other components
• specific features - usable in most of the processes, such as comment form or process history log
• custom business logic - related to a certain process or organization, a CRM customer search form would be an example for that.

The UI is provided by Vaadin, therefore its customization requires no HTML/CSS skills - it is programmed directly in Java. And with Vaadin, we can build beautiful, rich web interfaces with ease.

Enterprise integration

Aperte Workflow allows users to build automated steps as plugins. As with User Interface, some of them can be quite generic (e.g. invoke Drools), others - very specific. All of them can be reused in different processes without any trouble.

Also, Aperte Workflow comes with a sample plugin, that embeddes Mule ESB as an OSGi service. Other plugins can deploy simple Mule services, and processes can invoke them using Mule automated step. But - this is just a convienent sample of integration mechanisms and if for example ServiceMix would to be embedded as an integrated ESB - why not?

Extensible data model 

Aperte Workflow's data model is built with JPA/Hibernate. That grants us significant portability with little effort, but also allows the plugins to provide their own data model without performance downgrade - the data model is mapped directly to SQL database. 

BPMN2.0 modeler

When designing BPM system from scratch, around existing BPM engine or solution, you usually have to configure process in two places - one for the BPM engine and the other for your customizations. Rarely you would take the effort to create a graphical, coherent tool to do that. 

But, as Aperte Workflow is designed for more than one deployment, such effort pays off. That's why we have taken Signavio Core Components BPMN2.0 editor and combined it with our configuration controls into one, complete, browser-based process modeler.

How Aperte Workflow is built?

We have decided to build Aperte Workflow on top of existing, Open Source technologies. That not only saves us from the trouble of building everything from scratch, but also provides other, even more significant advantages:

• Maturity from the start. We don't have to test, mature and grow our components. They are already checked by thousands of their users and built around their feedback.
• Much easier to learn. It is much easier to start using or diagnose a problem with a solution built from technologies you are already familiar with, or that have a thriving community around them.
• New features. If you are using your own software, there is a great chance, that all the features have to analyzed, provided, tested and checked by you. 

Aperte Workflow is built from many Open Source libraries and frameworks, to mention only the most significant:

• Liferay Portal as a default JSR168/286 container
• jBPM and Activiti BPM engines - it is also possible to user other implementations
• Vaadin as a default framework for building user interfaces
• Apache Felix - an OSGi framework
• Hibernate - ORM/persistence
• Mule ESB - as a sample embedded ESB provider

Create once, use many times

Aperte Workflow is great when it comes to building reusable components. Yes, many processes may differ, but building blocks for them can be re-used many times. Some of them, such as simple form displaying/editing process attributes or comment input form can be universal across the whole world, other - can be universal only in a specific organization - for example CRM customer search form would be applicable to many processes but only in current organization.

As Aperte Workflow is created in Java(tm) technology, the natural and simplest way to provide plugin mechanisms is to use an OSGi framework. OSGi is great, because it provides a standarized, well-thought way to:

• isolate each plugin within its classloader space - with its own libraries and dependencies
• provide a means for plugins to communicate with each other
• manage dependencies between plugins

With Aperte Workflow, reusability is provided on many levels, in compliance with the spirit of Service Oriented Architecture. Aperte Workflow services (bundled in OSGi plugins) can provide for example:

• user interface components
• automated steps performing business logic or integration
• localization (I18N)
• custom mechanisms that access Aperte Workflow API - e.g. start processes or provide embedded ESB for processes
• process dictionaries

and many more..