{"componentChunkName":"component---src-templates-blog-post-js","path":"/kafka-consumer/","result":{"data":{"markdownRemark":{"html":"

No artigo Produzindo mensagens com Kafka vimos como podemos usar o Kafka para enviar mensagens e agora vamos ver como podemos receber essas mensagens de forma simples, sem dor de cabeça e com a vantagem de termos a segurança que o Schema Registry trás com a validação de contratos.

\n

Projeto

\n

Esse projeto consistirá em receber as mensagens no formato Avro do Kafka e irá enviar um email para o usuário.\nPara iniciar o projeto foi utilizado o Spring Boot mas sem a dependência do spring kafka. Nesse projeto utilizaremos o Spring para subir a aplicação mas queremos ter mais controle sobre os processos do Kafka.

\n

Criando projeto

\n

Segue o pom.xml do projeto com os plugins e dependências utilizadas:

\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n\txmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n\txsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd\">\n\t<modelVersion>4.0.0</modelVersion>\n\t<parent>\n\t\t<groupId>org.springframework.boot</groupId>\n\t\t<artifactId>spring-boot-starter-parent</artifactId>\n\t\t<version>2.4.1</version>\n\t\t<relativePath /> <!-- lookup parent from repository -->\n\t</parent>\n\t<groupId>com.irs</groupId>\n\t<artifactId>sender</artifactId>\n\t<version>0.0.1-SNAPSHOT</version>\n\t<name>sender email</name>\n\t<description>Send email</description>\n\n\t<properties>\n\t\t<java.version>11</java.version>\n\t</properties>\n\n\t<dependencies>\n\t\t<dependency>\n\t\t\t<groupId>org.springframework.boot</groupId>\n\t\t\t<artifactId>spring-boot-starter-mail</artifactId>\n\t\t</dependency>\n\n\t\t<dependency>\n\t\t\t<groupId>org.projectlombok</groupId>\n\t\t\t<artifactId>lombok</artifactId>\n\t\t\t<optional>true</optional>\n\t\t</dependency>\n\n\t\t<!-- https://mvnrepository.com/artifact/org.apache.avro/avro -->\n\t\t<dependency>\n\t\t\t<groupId>org.apache.avro</groupId>\n\t\t\t<artifactId>avro</artifactId>\n\t\t\t<version>1.10.1</version>\n\t\t</dependency>\n\n\t\t<!--dependencies needed for the kafka part -->\n\t\t<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->\n\t\t<dependency>\n\t\t\t<groupId>org.apache.kafka</groupId>\n\t\t\t<artifactId>kafka-clients</artifactId>\n\t\t</dependency>\n\n\t\t<!-- https://mvnrepository.com/artifact/io.confluent/kafka-avro-serializer -->\n\t\t<dependency>\n\t\t\t<groupId>io.confluent</groupId>\n\t\t\t<artifactId>kafka-avro-serializer</artifactId>\n\t\t\t<version>5.3.0</version>\n\t\t</dependency>\n\n\t\t<dependency>\n\t\t\t<groupId>org.springframework.boot</groupId>\n\t\t\t<artifactId>spring-boot-starter-mail</artifactId>\n\t\t</dependency>\n\n\t\t<dependency>\n\t\t\t<groupId>org.springframework.boot</groupId>\n\t\t\t<artifactId>spring-boot-configuration-processor</artifactId>\n\t\t\t<optional>true</optional>\n\t\t</dependency>\n\n\t\t<dependency>\n\t\t\t<groupId>org.springframework.boot</groupId>\n\t\t\t<artifactId>spring-boot-starter-test</artifactId>\n\t\t\t<scope>test</scope>\n\t\t</dependency>\n\t</dependencies>\n\n\t<build>\n\t\t<plugins>\n\t\t\t<plugin>\n\t\t\t\t<groupId>org.springframework.boot</groupId>\n\t\t\t\t<artifactId>spring-boot-maven-plugin</artifactId>\n\t\t\t</plugin>\n\n\t\t\t<plugin>\n\t\t\t\t<groupId>org.springframework.boot</groupId>\n\t\t\t\t<artifactId>spring-boot-maven-plugin</artifactId>\n\t\t\t</plugin>\n\t\t\t<!--for specific record -->\n\t\t\t<plugin>\n\t\t\t\t<groupId>org.apache.avro</groupId>\n\t\t\t\t<artifactId>avro-maven-plugin</artifactId>\n\t\t\t\t<version>1.10.1</version>\n\t\t\t\t<executions>\n\t\t\t\t\t<execution>\n\t\t\t\t\t\t<phase>generate-sources</phase>\n\t\t\t\t\t\t<goals>\n\t\t\t\t\t\t\t<goal>schema</goal>\n\t\t\t\t\t\t\t<goal>protocol</goal>\n\t\t\t\t\t\t\t<goal>idl-protocol</goal>\n\t\t\t\t\t\t</goals>\n\t\t\t\t\t\t<configuration>\n\t\t\t\t\t\t\t<sourceDirectory>${project.basedir}/src/main/resources/avro</sourceDirectory>\n\t\t\t\t\t\t\t<stringType>String</stringType>\n\t\t\t\t\t\t\t<createSetters>false</createSetters>\n\t\t\t\t\t\t\t<enableDecimalLogicalType>true</enableDecimalLogicalType>\n\t\t\t\t\t\t\t<fieldVisibility>private</fieldVisibility>\n\t\t\t\t\t\t</configuration>\n\t\t\t\t\t</execution>\n\t\t\t\t</executions>\n\t\t\t</plugin>\n\t\t\t<!--force discovery of generated classes -->\n\t\t\t<plugin>\n\t\t\t\t<groupId>org.codehaus.mojo</groupId>\n\t\t\t\t<artifactId>build-helper-maven-plugin</artifactId>\n\t\t\t\t<version>3.0.0</version>\n\t\t\t\t<executions>\n\t\t\t\t\t<execution>\n\t\t\t\t\t\t<id>add-source</id>\n\t\t\t\t\t\t<phase>generate-sources</phase>\n\t\t\t\t\t\t<goals>\n\t\t\t\t\t\t\t<goal>add-source</goal>\n\t\t\t\t\t\t</goals>\n\t\t\t\t\t\t<configuration>\n\t\t\t\t\t\t\t<sources>\n\t\t\t\t\t\t\t\t<source>target/generated-sources/avro</source>\n\t\t\t\t\t\t\t</sources>\n\t\t\t\t\t\t</configuration>\n\t\t\t\t\t</execution>\n\t\t\t\t</executions>\n\t\t\t</plugin>\n\t\t</plugins>\n\t</build>\n\n\t<repositories>\n\t\t<repository>\n\t\t\t<id>confluent</id>\n\t\t\t<url>https://packages.confluent.io/maven/</url>\n\t\t</repository>\n\t</repositories>\n\n</project>
\n

Atualizando Schema

\n

Antes de iniciar o projeto precisamos atualizar o nosso Schema pois na primeira versão não havia a informação de email e agora precisamos disso para que o serviço funcione corretamente, o novo Schema irá manter os campos que já possuía e será adicionado o campo de email:

\n
{\n     \"type\": \"record\",\n     \"namespace\": \"com.irs.register.avro.taxpayer\",\n     \"name\": \"TaxPayer\",\n     \"version\": \"2\",\n     \"fields\": [\n       { \"name\": \"name\", \"type\": \"string\", \"doc\": \"Name of TaxPayer\" },\n       { \"name\": \"document\", \"type\": \"string\", \"doc\": \"Document of TaxPayer\" },\n       { \"name\": \"email\", \"type\": \"string\", \"doc\": \"Email of TaxPayer\" },\n       { \"name\": \"situation\", \"type\": \"boolean\", \"default\": false, \"doc\": \"Legal situation of TaxPayer\" }\n     ]\n}
\n

Configurando consumidor

\n

Vamos começar as configurações de propriedades para quando a aplicação for se conectar ao Kafka:

\n
@Configuration\npublic class KafkaConfiguration implements MessageConfiguration<TaxPayer> {\n\t\n\t@Autowired\n\tprivate KafkaProperties kafkaProperties;\n\n\t@Bean(name = \"taxpayerConsumer\")\n\t@Override\n\tpublic KafkaConsumer<String, TaxPayer> configureConsumer() {\n\t\t\n\t\tProperties props = new Properties();\n\t\t\n\t\tprops.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getBootstrapServers());\n\t\tprops.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaProperties.getGroupId());\n\t\tprops.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, kafkaProperties.getAutoCommit());\n\t\tprops.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, kafkaProperties.getOffsetReset());\n\t\t\n\t\tprops.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, kafkaProperties.getKeyDesserializer());\n\t\tprops.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, kafkaProperties.getValueDesserializer());\n\t\t\n\t\tprops.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, kafkaProperties.getSchemaRegistryUrl());\n\t\tprops.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, kafkaProperties.isSpecificAvroReader());\n\t\t\n\t\treturn new KafkaConsumer<String, TaxPayer>(props);\n\t}\n\n}
\n

Basicamente configuramos as urls de conexão com o Kafka e Schema Registry, definimos o nosso group id para identificação do nosso consumidor no Consumer Group, também configuramos o Auto Commit do Offset juntamente com o Auto Offset Reset, que indica se queremos sempre buscar as mensagens desde o inicio, do último offset commitado e etc. e por fim definimos que queremos usar um leitor específico do nosso Avro com a configuração SPECIFICAVROREADER_CONFIG.

\n

Feito isso basta instanciarmos a classe KafkaConsumer passando as nossas props.

\n

Implementando o consumidor

\n

Com as configurações prontas agora basta que criemos a classe de serviço que irá utilizar o KafkaConsumer e irá buscar as mensagens no Kafka:

\n
@Service\n@Slf4j\npublic class KafkaConsumerService implements Consumer<TaxPayer> {\n\n\t@Autowired\n\t@Qualifier(\"taxpayerConsumer\")\n\tprivate KafkaConsumer<String, TaxPayer> kafkaConsumer;\n\n\t@Override\n\tpublic String topic() {\n\t\treturn \"taxpayer-avro\";\n\t}\n\n\t@PostConstruct\n\t@Override\n\tpublic void receive() {\n\n\t\tkafkaConsumer.subscribe(Collections.singleton(this.topic()));\n\n\t\twhile (true) {\n\n\t\t\ttry {\n\n\t\t\t\tkafkaConsumer.poll(Duration.ofMillis(1000)).forEach(record -> {\n\n\t\t\t\t\tlog.info(\"Recebendo TaxPayer\");\n\n\t\t\t\t\tTaxPayer taxpayer = record.value();\n\n\t\t\t\t\tPerson person = Person.builder().email(taxpayer.getEmail()).name(taxpayer.getName()).build();\n\n\t\t\t\t\tlog.info(person.toString());\n\n\t\t\t\t});\n\n\t\t\t\tkafkaConsumer.commitSync();\n\n\t\t\t} catch (Exception ex) {\n\t\t\t\tlog.error(\"Erro ao processar mensagem\", ex);\n\t\t\t}\n\n\t\t}\n\n\t}\n\n}
\n

Na classe KafkaConsumerService temos o método receive e nele usamos o KafkaConsumer para nos subscrevermos no tópico, que é o taxpayer-avro e entramos em um loop while para sempre ficarmos buscando as mensagens no Kafka e após isso commitamos o offset para as partições em caso de sucesso ou lançamos uma Exception em caso de falha.

\n

Serviço de Email

\n

Para configurar o serviço de envio de mensagem de email vamos usar a classe JavaMailSender que o Spring fornece:

\n
@Configuration\npublic class MailConfigurer {\n\t\n\t@Bean\n\tpublic JavaMailSender getJavaMailSender() {\n\t    JavaMailSenderImpl mailSender = new JavaMailSenderImpl();\n\t    mailSender.setHost(\"smtp.gmail.com\");\n\t    mailSender.setPort(587);\n\t    \n\t    mailSender.setUsername(\"fake@gmail.com\");\n\t    mailSender.setPassword(\"senha secreta\");\n\t    \n\t    Properties props = mailSender.getJavaMailProperties();\n\t    props.put(\"mail.transport.protocol\", \"smtp\");\n\t    props.put(\"mail.smtp.auth\", \"true\");\n\t    props.put(\"mail.smtp.starttls.enable\", \"true\");\n\t    props.put(\"mail.debug\", \"true\");\n\t    \n\t    return mailSender;\n\t}\n\n}
\n

Após isso podemos criar uma classe de serviço especializada em enviar os emails:

\n
@Service\n@Slf4j\npublic class EmailService implements Email{\n\t\n\t@Autowired\n    private JavaMailSender emailSender;\n\n\t@Override\n\tpublic void sendMessage(Person person) {\n\t\t\n\t\ttry {\n\t\t\t\n\t\t\tlog.info(\"Try to send email for : \" + person.getEmail());\n            SimpleMailMessage message = new SimpleMailMessage();\n            message.setFrom(\"noreply@irs.com\");\n            message.setTo(person.getEmail());\n            message.setSubject(\"Confirmação de recebimento\");\n            message.setText(person.getName() + \" seus dados foram recebidos com sucesso\");\n\n            emailSender.send(message);\n            \n        } catch (MailException ex) {\n        \tlog.error(\"Error to send email\", ex);\n        }\n\t\t\n\t}\n\n}
\n

O método sendMessage recebe um objeto do tipo Person e envia o email com a nossa mensagem.

\n

Agora podemos injetar a classe de email no consumidor para enviar o email a cada mensagem recebida:

\n
@Service\n@Slf4j\npublic class KafkaConsumerService implements Consumer<TaxPayer> {\n\n\t@Autowired\n\t@Qualifier(\"taxpayerConsumer\")\n\tprivate KafkaConsumer<String, TaxPayer> kafkaConsumer;\n\n\t@Autowired\n\tprivate Email email;\n\n\t@Override\n\tpublic String topic() {\n\t\treturn \"taxpayer-avro\";\n\t}\n\n\t@PostConstruct\n\t@Override\n\tpublic void receive() {\n\n\t\tkafkaConsumer.subscribe(Collections.singleton(this.topic()));\n\n\t\twhile (true) {\n\n\t\t\ttry {\n\n\t\t\t\tkafkaConsumer.poll(Duration.ofMillis(1000)).forEach(record -> {\n\n\t\t\t\t\tlog.info(\"Recebendo TaxPayer\");\n\n\t\t\t\t\tTaxPayer taxpayer = record.value();\n\n\t\t\t\t\tPerson person = Person.builder().email(taxpayer.getEmail()).name(taxpayer.getName()).build();\n\n\t\t\t\t\temail.sendMessage(person);\n\n\t\t\t\t});\n\n\t\t\t\tkafkaConsumer.commitSync();\n\n\t\t\t} catch (Exception ex) {\n\t\t\t\tlog.error(\"Erro ao processar mensagem\", ex);\n\t\t\t}\n\n\t\t}\n\n\t}\n\n}
\n

Agora a cada mensagem que for recebida ele irá enviar um email.

\n

Código fonte

\n

O código desse projeto se encontra no GitHub

\n

Alternativa com Spring Boot

\n

No exemplo desse artigo foi mostrado como consumir as mensagens do Kafka sem utilizar as libs do Spring Bot, caso tenha interesse em ver como seria a configuração de um consumidor utilizando a implementação do Spring Boot esses dois artigos mostram como fazer isso, a primeira usando annotations e outra sem usar annotations:

\n","fields":{"slug":"/kafka-consumer/"},"frontmatter":{"date":"18 de janeiro de 2021","description":"Criando um consumidor de mensagens com Kafka e Java","title":"Consumindo mensagens do Kafka sem dor de cabeça","tags":["Kafka","Consumer","Schema Registry","Java"],"author":"Guilherme Alves","image":{"id":"f10552b3-666c-547b-bb9b-025cac5ee94e","publicURL":"/static/26f7480e399fb3d5b5522e597e936f64/kafka-consumer.png","childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAAAsSAAALEgHS3X78AAAC70lEQVQ4y21U3UtTYRg//4DQRd0IXnnjfdCVYoHl+hJ1uqYS1PwqLcGPKzfELmozm1tLGWPBhKALQYP5hUtImVAqbgzmYnPOUcO5prl0H+3z1/u+65yN6IHDeZ7nPO/vPL/n+Z3DgVgulwNv/uAJdtzfsOXy43vop5CnNcV1vH90dMTuJpMJy8vL4LLZLEtE47/xZfcALn8Q59Eo0tkcdg+CBPgAiWRKAAqFQtDr9UKs0WigVCpRXl4OtVoNjn/w2bmPRCoLv28fL8fGcHpKu8shHDnH9le/0NEPAshxHOSjoyy22WwsLisrw8DAALh9rxcutxcObyBfYLejv78fcrkc5vl5ltt2+fArGmf+aSSCCyUluEhA7Ds2hMNhBlhaWgqj0QhOMzGBmtpbmFtZZwesGxsYHh6GSqXCYTDIcnvBCKGdZr6P5J5fvQZfySUMSu7BurkJ/dQUPB4PVldX85RXLB8J6E280b1GR0cnqqurIRKJ0NraCvX4OLp6++DZ8zLAwMkJnl2+gq1GCT7MzWFwaAjFxvFzaZK2wfTuPW5cr4GothZaMuzensfQG024Wy9G8PBQYGB4a8xvmlw6nQ6xeH4cmUwGXObvlrse9eD+g3Y0iptxu64e9WIJ8ZsglrRArhhhNalUCjKZDIuLi0JHn9bW4Ha7mU8Vw6XT+dlMT09DpXxBQGWoa27DHXELHrZ3ou/pE1gsFlaTTCbR3d0Nq9Va0K3fj6WlpUKHvA5dLheqqqoYVfWrcWgm1Ew+arK0s7MzVpNIJFBZWQmHwyEAxmIxzM7OCrLieIfSoQLVarVsLlSkU2R79AoEAgKlkZERxMnMqP4MBgMoQwp4fHxcWAptlZrT6URFRQXW1/MSkkqljCL/Qmpms5m9kIJQVvNEq5OTk0yPQoc8bS8ReUNDA9MUNYVCwfRIjZ81ndfCwoJA2U4+hJmZmYJseMrFH/7/Yp5ysV8c82e44sP0TunzMX+o+E9Dn/Mj+reG2h/dGupbd33jigAAAABJRU5ErkJggg==","aspectRatio":1.3445378151260505,"src":"/static/26f7480e399fb3d5b5522e597e936f64/761b8/kafka-consumer.png","srcSet":"/static/26f7480e399fb3d5b5522e597e936f64/c67b7/kafka-consumer.png 320w,\n/static/26f7480e399fb3d5b5522e597e936f64/761b8/kafka-consumer.png 525w","sizes":"(max-width: 525px) 100vw, 525px"}}}},"timeToRead":8}},"pageContext":{"slug":"/kafka-consumer/","previous":{"fields":{"slug":"/flyway-java/"},"frontmatter":{"title":"Versionando o banco de dados com Flyway","date":"11 jan, 2021"}},"next":{"fields":{"slug":"/spring-validacao/"},"frontmatter":{"title":"Validando requisições com Spring Boot","date":"26 jan, 2021"}}}},"staticQueryHashes":["63159454"]}