Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

January 22 2018

ProcessOne: Using TLS Authentication for your Go Kafka Client

If you want to access a Kafka server that have enabled TLS, you will need to be able to use certificate to connect from your Sarama / Go client. This article outlines the needed steps to configure it properly.

Configuring your Kafka server to support authentication

If you are managing your own Kafka service and would like to enable authentication, you should read this article from Confluent documentation site: Encryption and Authentication using SSL.

Converting Java keystore and truststore

The first steps to easily handle your certificates from Go is to convert them to a set of PEM files.

Here are the commands to extract the Certificate Authority (CA) certificate:

$ keytool -importkeystore -srckeystore kafka.server.truststore.jks -destkeystore server.p12 -deststoretype PKCS12
$ openssl pkcs12 -in server.p12 -nokeys -out server.cer.pem

You can then convert your client keystore to be usable from Go, with similar commands:

$ keytool -importkeystore -srckeystore kafka.server.keystore.jks -destkeystore client.p12 -deststoretype PKCS12
$ openssl pkcs12 -in client.p12 -nokeys -out client.cer.pem
$ openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key.pem

Go Kafka client supporting TLS authentication

To connect to the server and authenticate with TLS, you just need to generate the proper TLSConfig. Here is the relevant code:

func NewTLSConfig(clientCertFile, clientKeyFile, caCertFile string) (*tls.Config, error) {
 tlsConfig := tls.Config{}

 // Load client cert
 cert, err := tls.LoadX509KeyPair(clientCertFile, clientKeyFile)
 if err != nil {
  return &tlsConfig, err
 }
 tlsConfig.Certificates = []tls.Certificate{cert}

 // Load CA cert
 caCert, err := ioutil.ReadFile(caCertFile)
 if err != nil {
  return &tlsConfig, err
 }
 caCertPool := x509.NewCertPool()
 caCertPool.AppendCertsFromPEM(caCert)
 tlsConfig.RootCAs = caCertPool

 tlsConfig.BuildNameToCertificate()
 return &tlsConfig, err
}

The code is then extremely simple to connect:

func main() {
 tlsConfig, err := NewTLSConfig("bundle/client.cer.pem",
  "bundle/client.key.pem",
  "bundle/server.cer.pem")
 if err != nil {
  log.Fatal(err)
 }
 // This can be used on test server if domain does not match cert:
 // tlsConfig.InsecureSkipVerify = true

 consumerConfig := sarama.NewConfig()
 consumerConfig.Net.TLS.Enable = true
 consumerConfig.Net.TLS.Config = tlsConfig

 client, err := sarama.NewClient([]string{"localhost:9093"}, consumerConfig)
 if err != nil {
  log.Fatalf("unable to create kafka client: %q", err)
 }

 consumer, err := sarama.NewConsumerFromClient(client)
 if err != nil {
  log.Fatal(err)
 }
 defer consumer.Close()

 consumerLoop(consumer, "mytopic")
}

The consumerLoop has nothing special regarding TLS authentication. You can just use your standard Sarama code. You can read the full code on Github: base-client.go.

January 18 2018

Monal IM: Next Mac Beta is up

Another Mac beta is up.  I have fixed all the crashes that came in on the last beta.

JC Brand: Converse 3.3 has been released

Last night I released version 3.3.0 of Converse.js, and as often happens with big releases, I made a quick bugfix release (3.3.1) today.

The bugfix release turns off some CSS3 animations for new messages which caused degraded performance on Firefox. On Chrome the animations render smoothly, so if you'd like you can still turn them on with the show_message_load_animation config option.

What's in the release?

Maintaining a long-term open source front-end JavaScript library almost feels like a Sisyphean task sometimes. As soon as you've rolled the big stone up the hill, the whole JS ecosystem, best practices and tooling changes and you find yourself at the bottom of the hill again.

This release is therefore heavy on changes under the hood, with the aim of modernizing and improving the quality of the code.

Besides that, I also spent time squashing lots of small usability bugs and on improving performance.

Converse.js now uses a Virtual DOM

Various views, such as the registration form, are now rendered by means of a Virtual DOM. I wrote a new Backbone view called Backbone.VDOMView for this, and blogged about it here: https://opkode.com/blog/backbone-vdomview/

No more jQuery

Looking at the git log, I started rewriting code to not use jQuery in January 2017.

So this change has been a year in the works. I often asked myself whether I should spend time on this and not rather do something else, like adding new features, especially since removing jQuery has taken a lot of time.

However, there were some good reasons, or perhaps motivations, for me to remove jQuery.

Before ES6 promises were available, I used $.Deferred. However, jQuery's deferreds weren't compatible with Promises, so when ES6 Promises came around, I had to rewrite lots of code to use Promises.

I used $.show and $.hide quite a bit, and then it turned out that the way jQuery was doing it (by adding/removing display: none to the DOM element) is not good practice and also very slow.

So I started writing my own utility functions to replace jQuery's.

The last straw for me was when jQuery 3 came out, and half of Converse.js's ~240 tests failed once I plugged it in.

After spending some time trying to figure out what backward incompatible changes they made and how I should update the code, I decided to instead rip jQuery out entirely.

It's still used in the tests, but it's no longer included in any build.

Since removing it, I've noticed a remarkable reduction in time to run the tests.

By looking at how quickly the tests run now, the code seems to run much faster without jQuery.

Less weight

Besides removing jQuery, I also updated Converse.js to load translations at runtime, and only the exact translation JSON file that's needed.

Due to these changes, the unminified built has shrunk from 3.38MB to 2.8MB, and the minified build from 1.66MB to 1.2MB.

And this is while adding the virtual DOM code.

Route to a specific chat room via the URL

It's now possible to directly link to a specific chat room, or to the registration page (instead of the login page) via the URL.

For example, the URL https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org will take you immediately to the Converse.js "Discuss" chat room, once you've logged in.

What else?

Lots of other bugfixes and improvements have been added in this release. For more details, check the changelog.

Notable absent from this release are some desired features, such as file sharing, message corrections, message receipts and the like.

I would love to add some of these often requested features, however I had to get the house in order so to speak, by modernizing the code and squashing lots and lots of little usability and performance bugs.

That said, Converse.js takes up a LOT of my free time and not a single line of code in this release was paid for.

If you or your company make use of converse.js, please consider sponsoring it on Patreon or Liberapay.

Thanks

Thanks goes out to everyone who's made pull requests and bug reports over the last months.

And thanks also to the folks who hang out in the Converse.js Discusss chat room and who have there provided valuable feedback.

January 16 2018

Monal IM: New Mac beta up

There is a new beta of the Mac client with all of the recent  fixes and addition of group messaging.  I know there are still a few issues, for example it doesn’t save the password for auto joining a bookmarked group. Im curious to see what other issues people encounter. Its been working ok for me the last couple of days as I’ve been using it to lurk on the XMPP developers group chats.

Paul Schaub: Smack: Some busy nights

Hello everyone!

This weekend I stayed up late almost every evening. Thus I decided that I wanted to code something, but I wasn’t sure what, so I took a look at the list of published XEPs to maybe find something that is easy to implement, but missing from Smack.

I found that XEP-0394: Message Markup was missing from Smacks list of supported extensions, so I began to code. The next day I finished my work and created Smack#194. One or two nights later I again stayed up late and decided to take another look for an unimplemented XEP. I settled on XEP-0382: Spoiler Messages  this time, which was really easy to implement (apart from the one little attribute, which for whatever reason I struggled to parse until I found a solution). The result of that night is Smack#195.

So if you find yourself laying awake one night with no chance to sleep, just look out for an easy to do task on your favourite free software project. I’m sure this will help you sleep better once the task is done.

Happy Hacking!
Vanitasvitae

Monal IM: The new group chat UI

I’m getting closer to the UI that I would like for group chats. It might not be the greatest thing for power users but I suspect it will work well for most people. Especially those familiar with iMessage or other chat clients already.   Things to note in the picture below, proper nick name support and subjects. I’ve decided to treat the subject like the group name.  I’ve observed that in other chat clients people like to change the group name on a whim mid conversation.  This is not something the muc spec supports as far as I can tell. People  can however change the subject, so subject it shall be.

January 12 2018

Monal IM: Mac has group chat

The Mac client has always had group chat but didn’t have a UI surfacing the functionality. I am adding that now.  I know favorites and auto join have long been asked for. I hope to have that in the iOS client as well. 

ProcessOne: ejabberd 18.01

ejabberd 18.01 is a bugfix release. This version of ejabberd Community Server is a good candidate for Linux distributions packaging as it concludes a year of development and stabilised all recent changes for production use.

Fixes

  • Fix TLS driver memory management
  • Fix privacy_set command
  • Report ‘fs’ support as unavailable on SunOS
  • Let mod_block_strangers bounce an error when a message is rejected

Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

January 10 2018

Paul Schaub: Reworking smack-omemo

A bit over a year ago I started working on smack-omemo as part of my bachelor thesis. Looking back at the past year, I can say there could have hardly been a better topic for my thesis. Working with Smack brought me deep into the XMPP world, got me in contact with a lot of cool people and taught me a lot. Especially the past Google Summer of Code improved my skills substantially. During said event, I took a break from working on smack-omemo, while focussing on a Jingle implementation instead. After the 3 months were over, I dedicated my time to smack-omemo again and realized, that there were some points that needed improvements.

One major issue was, that my “OmemoStore” class, which is responsible for storing keys, sessions, etc. was not having access to the users data before the user logged in. The reason for that is, that my implementation allows multiple OMEMO instances to be running on the same connection. That requires the OmemoStore to store keys for multiple instances (devices), which I distinguished based on the Jid and deviceId of the user. The problem here is, that the Jid is unknown before the user logged in (they might use a burner jid for example, or use an authentication system with username and password which differ from the jid).

While this is an edgecase, it lead to issues. I implemented a workaround for that problem (using the username instead of BareJid in case the connection is not authenticated), which caused numerous problems.

I thought about replacing the Jid as an identifier with something else, but nothing was suitable, so I started a major rework of the implementation as a whole. One important aspect I wanted to preserve is that smack-omemo should still be somewhat usable even when the connection is not authenticated (ie. the user should still be able to scan qr codes and make trust decisions).

The result of my work (so far) is a diff of “+6,300 −5,361″, and a modified API (sorry to all those who already use smack-omemo :O). One major change is, that the OmemoStore no longer stores trust decisions. Instead those decisions are now made by the client itself, who must implement a OmemoTrustCallback. That way trust decisions can be made while the OmemoManager is offline. Everything else what remained in the OmemoStore is only needed when the connection is authenticated and messages are received.

Furthermore I got rid of the OmemoSession class. Session handling is done in libsignal already, so why would I want to have a session related class as well (especially since libsignal doesn’t give you any feedback about what happens with the session, so you have to keep sessions in sync manually)? I recommend everyone who wants to implement OMEMO themselves not to create a “OmemoSession” class and instead rely on libsignals session management.

OMEMO sessions are somewhat brittle. You can never know, whether a recipient received your message, or if it failed to decrypt for some reason. There is no signalling to provide feedback about the sessions state. Because of the fact that even message encryption can go wrong, the old API was very ugly. Originally I first checked, whether there are devices which still need a trust decision to be made and threw an exception if that was the case. Then I tried to build sessions for devices without session and threw an exception when session negotiation failed. Then I tried to encrypt the message for all recipients and threw an exception if something went wrong… Oh and the exception I threw when sessions could not be negotiated contained a list of all devices with intact sessions, so the user could retry to encrypt the message, only for all devices which had a session.

Ugly!!!

The new API is much cleaner. I still throw an exception when there are undecided devices, but otherwise I always return an OmemoMessage object. That object has a map of OmemoDevices for which message encryption failed, alongside the respective exceptions, so the client can check if and what went wrong.

Also sessions are now “completed” whenever a preKeyMessage arrives.
Prior to this change it could happen, that two senders chose the same PreKey from a bundle in order to create a session. That could cause on of both session to break which lead to message loss. Now whenever smack-omemo receives a preKeyMessage, it instantly responds with an empty message to make the session stable.
This was proposed by Philipp Hörist.

Other changes include a new OmemoStore implementation, the CachingOmemoStore, which can either wrap other OmemoStores to provide a caching layer, or can be used standalone as an ephemeral store for testing purposes.

Also the integration tests were improved and are much simpler and more readable now.

All in all the code got much cleaner now and I hope that at some point it will be audited to find all the bugs I oversaw :D (everyone who wants to take a look for themselves, the code can currently be found at Smacks Repository. I’m always thankful for any types of feedback)

I hope this changes will make it to Smack 4.2.3, even though here are still some things I have to do, but all in all I’m already pretty satisfied with how smack-omemo turned out so far.

Happy Hacking!

Monal IM: Monal OSX 2 beta 7 out

OSX Beta 7 has been pushed  out. This version support making and receiving Jingle calls.  I have also added Monal to brew. You can now do this:

brew cask install monal

 

January 09 2018

ProcessOne: ejabberd 2017 year in review

This was an amazing year for ejabberd! With almost regular monthly updates, tons of new features and improvements, we worked hard to make ejabberd the best XMPP server available. Let’s take a look at the past year, and let us wish you a Happy New Year!

January – 17.01

We started last year with a cleanup. Most notably, we introduced SSL in PostgreSQL connections and improved database migration.

March – 17.03

This release introduced dynamic configuration reload, to ease ejabberd administration and management. To ease end users’ life, we created mod_block_strangers to significantly reduce XMPP spam.

April – 17.04

One of the most significant features of 2017 was introduced in April: Redis and SQL backends can now be used to store RAM tables, just like Mnesia.

May – GSoC

In May, ejabberd was yet again accepted to Google Summer of Code ’17. With two projects, Support for “Let’s Encrypt” ACME protocol and Server-to-Server stream management support it was looking like a super-interesting summer.

June – 17.06

Another significant ejabberd update happend in the middle of 2017. This time, we introduced a handy Certificate Manager, new caching system and the highly demanded Riak support.

August – 17.08

At the end of summer we produced another ejabberd release that improved the XEP-0357: Push Notifications support, as well as made it easier to configure a cluster backend – although only for Mnesia, for now.

September – 17.09

This was a busy month for ProcessOne. The result of the two ejabberd GSoC projects was very satisfying, both for us and the participants. Additionally, this month’s ejabberd release introduced a new, much better avatar module, implemented XEP-0368, and updated OpenSSL libraries in our installer.

November – 17.11

Happy Birthday ejabberd! It’s been 15 years :) Apart from the celebrations, this month we introduced “Let’s Encrypt” ACME support created during ejabberd GSoC, and implemented much demanded PubSub improvements.

December – 17.12

With the end of this amazing year for ejabberd, we were happy to announce a major XMPP/PubSub project migration to ejabberd server. And with another successful ejabberd release, we introduced SNI for inbound connections, and improved support for PubSub v1.14 and OMEMO.

Conclusion

2017 was an amazing year for ejabberd, but we are not stopping there! Looking forward, we are happy to announce that ejabberd is compatible with XEP-0387: XMPP Compliance Suites 2018 as well as passes the Conversations Compliance Suite.

We have many new features and improvements lined up for this year. Look forward to our next release! To stay tuned, be sure to follow us on Twitter, Facebook, or subscribe to ejabberd newsletter.

Monal IM: Mac Jingle Voice Calls Work

Tested out jingle voice calls on the Mac today.  There is a call button on the toolbar but you can’t initiate calls quite yet. You can however, accept a call, talk and hang up.  Monal is the bare bones ui with the hang up button :).  Try it out with the beta that is currently available. 

January 08 2018

Monal IM: Revisiting Muc

Muc aka Group Chat has been something that’s been in Monal since the beginning.   I’ve decided to throw out all the old  UI for this and remake it.   The old version was something I wrote before I had very many users and based on my experience using pidgin/adium/gaim. It doesn’t work anywhere near as nice as I would like it to. Also given the greater importance of group conversations these days, I have decided make it a top level item.   There isn’t a lot of room for things on the tab bar at the bottom.  I think this is what the current four will be.  

Monal IM: iOS Voip works again and logs

Fixing the issues with Jingle Voip was a high priority this week.  The iOS client audio  should not have issues anymore.  I was testing this with pidgin (and wire shark) in a linux VM until I discovered Jitsi, which is a nice client in it’s own right.  I will likely test Voip with this going forward.

Continuing on to the next stop of the  refactoring train, I looked at some persistent bugs with the chat logs. It seems  I hadn’t made changes to that code since 2014.  It helps that Boston is experiencing record breaking cold right now, there isn’t a lot of motivation to go out.

The settings screen now has a chat logs option. It works like the old screens but with out the (null) entries.

 

 

January 07 2018

Tigase Blog: IoT over XMPP

Just over a year ago, Tigase presented a talk about IoT over XMPP at FOSDEM 2017. Our idea was to use an XMPP server as a go-between the user and the controlled device, eliminating direct access to devices from the internet.

January 05 2018

Monal IM: Jingle all the way

I’ve revisited of the jingle code that appears to have broken in the last couple of years. While I was at it I added jingle support to the Mac for audio calls.  I still need to write the audio code for OSX but the signaling component is all exactly the same as iOS. 

 

January 04 2018

Fanout Blog: The Edge is Nothing Without the Fog

Edge computing is hot right now. The growing maturity of IoT networks ranging from industrial to VR applications means that there’s an enormous amount of discussion around moving from the cloud to the edge (from us as well). But edge computing is only the first step.

...

January 03 2018

Peter Saint-Andre: Scales and Modes and Tetrachords, Oh My!

In preparation for recording my arrangements of music by Yes for solo electric bass, I've started taking music lessons with Mark Stefaniw, a fine bassist in the Denver area. Under Mark's tutelage I've been digging into music theory, which is both fascinating and enlightening (especially given my many years of musical learning at the surface)....

January 01 2018

Arnaud Joset: Errol: XMPP Automatic file sender

Errol is a file sender that rely on inotify. It can be used to watch a directory and automatically transfers the new files (or modified ones) with XMPP.

The origins

Errol find its origin in tasks I am doing for a small association, "Les compagnons du CEP", a joint buying organization who sells wines. I manage their ERP, the excellent Odoo. One of these tasks is the generation of their price list from an Excel spreadsheet (yeah, I know). I designed this process with a LaTeX generator written in python because I am fluent with it since 10 years. As I did not want to install a LaTeX distribution on the production server, the logical decision was to delocalize this task on another machine. The user uploads his excel file on a webpage, the file is saved in a "watched" directory and its transfer is triggered on the second machine with XMPP. The generation of the latex document and its compilation is performed with LaTeX and the resulting PDF is sent back to the server. The PDF is therefore available for download.

Why Errol?

In the fictional universe of Harry Potter, Errol is the Weasley family's owl. It is quite old and awkward. One could says the same about XMPP but Errol is quite useful, XMPP is too :-). Errol is a great grey owl. (see pictures)

photo credit: blurred.ca https://www.flickr.com/photos/blurredca/10527590684/ Photo credit: blurred.ca, Great Grey Owl


Howto

Prerequisites

Errol needs the following requirements:

  • A system supporting inotify (Linux).
  • an XMPP (jabber) account supporting the following XEPs: Stream Management, Publish-Subscribe, Multi-User Chat
  • A PubSub service where the nodes can be set as open. The node name is defined in the configuration file. I personally use sat_pubsub. A PubSub component developed for the project Salut à Toi.
  • A Multi-User Chat because not all XMPP accounts support PubSub. For now, some information are still shared through MUC messages. This behavior could change in the future.
  • The latest (dev) version of Slixmpp.

You can use your own XMPP server or choose a XMPP service among the following list.

Create the PubSub node

This step is optional if you already have a write access on the pubsub node. The following example use jp, the Salut à  Toi command-line interface but slixmpp or sleekxmpp can be used.

$ jp pubsub node create -f publish_model open be.agayon.errol:0 -s pubsub.agayon.be -c

The node name be.agayon.errol:0 is recommended in order to identify the functionality.

As an example, there are the node options on the service pubsub.agayon.be:

$ jp pubsub node info be.agayon.errol:0 -s pubsub.agayon.be
persist_items: True
deliver_payloads: True
serial_ids: False
publish_model: open
access_model: open
send_last_published_item: on_sub

If your server supports Personal Eventing Protocol(PEP) or if you do not want to use the generic PubSub service of your server, you can use your jid.

$ jp pubsub node create -f publish_model open be.agayon.errol:0 -s info@agayon.be -c

Tests

You can test your setup with the examples scripts of slixmpp.

Example:

./s5b_file_sender.py -j jid@example.org -p pass -r john@example.org -f /path/to/file.txt 

See the scripts for more information.

Getting started with Errol

First you need to clone the repository. Errol needs the following dependencies:

Installing

You can easily install errol with pip:

$ pip install errol

Note: errol can be installed in a virtualenv.

You can also clone the git repository:

$ git clone https://gitlab.com/jnanar/errol.git
$ cd errol
$ python3 setup.py install

On Archlinux:

A PKGBUILD will be available soon.

Configuration

You need to provide information about the XMPP account.

$ cat config.example.ini

[XMPP]
pubsub=pubsub.example.org
node=be.agayon.errol:0
room=chat@chat.example.org
jid=jid@example.org/errol
password=pass
ressource_receiver=-receiver
ressource_sender=-
nick_sender=example_sender
nick_receiver=example_receiver
receiver=jid@example.org/errol-receiver
  • jid : the jabber account
  • password: the xmpp password
  • pubsub: the PubSub server (publish activity)
  • room: the MUC (chatroom) where errol display information.

The files will be sent by jid@example.org/errol-0 and received by jid@example.org/errol-receiver . The nicks are the usernames used on the MUC.

Flying

Photo credit: Wisconsin Department of Natural Resources https://www.flickr.com/photos/widnr/ Photo credit: Wisconsin Department of Natural Resources, Great Grey Owl at Mauston

Once installed, Errol can be launched in a terminal.

$ errol --help
usage: errol [-h] [-e EVENTS] [-f FILE] [-d] -p PATH -c COMMAND

Automatic XMPP file sender and directory watcher

optional arguments:
  -h, --help            show this help message and exit
  -e EVENTS, --events EVENTS
                        Number of events to watch (delete, create modify) in
                        the directory. Once reached, the program stops.
  -f FILE, --file FILE  Config file containing XMPP parameters
  -d, --debug           set logging to DEBUG
  -p PATH, --path PATH  The path watched.
  -c COMMAND, --command COMMAND
                        The executed command: xmpp or watcher

In Hogwarts

If you want to watch the directory /tmp/sender, the following command can be used:

$ errol -f config.example.ini -p /tmp/sender -c watcher

All modified or new files created in the watched location will be sent by XMPP.

In Azkaban

If you want to receive the files, you have to launch Errol with the following command line.

$ errol -f config.example.ini -p /tmp/receiver -c xmpp

All the received files will be stored in the directory defined with the option '-p'.

License

This project is licensed under the GPLv3 - see the LICENSE.txt file for details

Why not X or Y?

Photo credit: Bernard Spragg. NZ https://www.flickr.com/photos/volvob12b/37310719232/ Photo credit: Bernard Spragg. NZ, Great Grey Owl (Strix nebulosa)

There are plenty solutions for this kind of needs. Some of them are more mature. I choose XMPP for several reasons:

  • agayon.be already provides a up-to-date XMPP server with all the needed XEPs enabled.
  • I do not want to open additional port on the client that performs the LaTeX compilation.
  • I wanted to learn to work with XMPP for machine to machine communications and use PubSub notifications (because why not?).

Among the alternatives, I could have build the service on top of:

  • sockets
  • HTTP file transfer
  • REST API
  • SSH and remote commands.
  • rsync
  • ...

Acknowledgments

Photo credit: lasta29, Great grey owl, Osaka Tennoji Zoo https://www.flickr.com/photos/115391424@N05/36873334554/ Photo credit: lasta29, Great grey owl, Osaka Tennoji Zoo

Links

December 31 2017

Monal IM: Notifications Screen

Push notifications have a lot of moving parts. To help people figure out where things may not be working (and to make sure everything is enabled), there is a new push notifications screen.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl