Authors: Jiayingm Lan, Lianglun Cheng, Guoheng Huang, Chi-Man Pun, Xiaochen Yuan, Shangyu Lai, Hongrui Liu, Wing-Kuen Ling

Abstract:

Multimodal image-to-image translation has received great attention due to its flexibility and practicality. The existing methods lack the generality of effective style representation, and cannot capture different levels of stylistic semantic information from cross-domain images. Besides, they ignore the parallelism for cross-domain image generation, and their generator can only be responsible for specific domains. To address these issues, we propose a novel Single Cross-domain Semantic Guidance Network (SCSG-Net) for coarse-to-fine semantically controllable multimodal image translation. Images from different domains are mapped to a unified visual semantic latent space by a dual sparse feature pyramid encoder, and then the generative module generates the result images by extracting semantic style representation from the input images in a self-supervised manner guided by adaptive discrimination. Especially, our SCSG-Net meets the needs of users in different styles as well as diverse scenarios. Extensive experiments on different benchmark datasets show that our method outperforms other state-of-the-art methods both quantitatively and qualitatively.

Authors: Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro IDE

Abstract:

Most content summarization models from the field of natural language processing summarize the textual contents of a collection of documents or paragraphs. In contrast, summarizing the visual contents of a collection of images has not been researched to this extent. In this paper, we present a framework for summarizing the visual contents of an image collection. The key idea is to collect the scene graphs for all images in the image collection, create a combined representation, and then generate a visually summarizing caption using a scene-graph captioning model. Note that this aims to summarize common contents across all images in a single caption rather than describing each image individually. After aggregating all the scene graphs of an image collection into a single scene graph, we normalize it by using an additional concept generalization component. This component selects the common concept in each sub-graph with ConceptNet based on word embedding techniques. Lastly, we refine the captioning results by replacing a specific noun phrase with a common concept from the concept generalization component to improve the captioning results. We constructed a dataset for this task based on the MS-COCO dataset using techniques from image classification and image-caption retrieval. Evaluation of the proposed method on this dataset showed promising performance.

Authors: Jianghai Wang, Menghao Hu; Yaguang Song, Xiaoshan Yang

Abstract:

Health-oriented food analysis has become a research hotspot in recent years because it can help people keep away from unhealthy diets. Significant progress has been made in recipe retrieval, food recommendation, nutrition and calorie estimation. However, existing works still cannot well balance the individual preference and the health. Multimodal food question and answering (Q&A) has a great potential in practical applications, but it is still not well studied. In this paper, we build a health-oriented multimodal food Q&A dataset (MFQA) with 9K question and answer pairs based on a multimodal food knowledge graph collected from a food-sharing website. In addition, we propose a knowledge-based multimodal food Q&A framework, which consists of three important parts: encoder module, retrieval module, and answer module. The encoder module can complete the encoding of text, vision and knowledge graph. The retrieval module is to filter out the most relevant knowledge from the knowledge graph. The answer module is responsible for analyzing the multimodal information in the query and relevant knowledge and predicting the correct answer. Extensive experimental results on the MFQA dataset demonstrate the effectiveness of our method. The code and dataset are available at https://anonymous.4open.science/r/HMFQA-861D.

Authors: Golsa Tahmasebzadeh, Eric Müller-Budack, Sherzod Hakimov, Ralph Ewerth

Abstract:

The consumption of news has changed significantly as the Web has become the most influential medium for information. To analyze and contextualize the large amount of news published every day, the geographic focus of an article is an important aspect in order to enable content-based news retrieval. There are methods and datasets for geolocation estimation from text or photos, but they are typically considered as separate tasks. However, the photo might lack geographical cues and text can include multiple locations, making it challenging to recognize the focus location using a single modality. In this paper, a novel dataset called Multimodal Focus Location of News (MM-Locate-News) is introduced. We evaluate state-of-the-art methods on the new benchmark dataset and suggest novel models to predict the focus location of news using both textual and image content. The experimental results show that the multimodal model outperforms unimodal models.

Authors: Yibo Hu, Chenyu Cao, Fangtao Li, Chenghao Yan, Jinsheng Qi, Bin Wu

Abstract:

Recognizing social relationships between multiple characters from videos can enable intelligent systems to serve human society better. Previous studies mainly focus on the still image to classify the relationships while ignoring the important data source of the video. With the prosperity of multimedia, the methods of video-based social relationship recognition gradually emerge. However, those methods either only focus on the logical reasoning between multiple characters or only on the direct interaction in each character pair. To that end, inspired by the rules of interpersonal social communication, we propose Overall-Distinctive GCN (OD-GCN) to recognize the relationships of multiple characters in the videos. Specifically, we first construct an overall-level character heterogeneous graph with two types of edges and rich node representation features to capture the implicit relationship of all characters. Then, we design an attention module to find mentioned nodes for each character pair from feature sequences fused with temporal information. Further, we build distinctive-level graphs to focus on the interaction between two characters. Finally, we integrate multimodal global features to classify relationships. We conduct the experiments on the MovieGraphs dataset and validate the effectiveness of our proposed model.

Authors: Haoran Ren, Hao Ren, Hong Lu, Cheng Jin

Abstract:

The weakly-supervised temporal action localization task aims to train a model that can accurately locate each action instance in the video using only video-level class labels. The existing methods take into account the information of different modalities (primarily RGB and Flow), and present numerous multi-modal complementary methods. RGB features are obtained by calculating appearance information, which are easy to be disrupted by the background. On the contrary, Flow features are obtained by calculating motion information, which are usually less disrupted by the background. Based on this phenomenon, we propose a Regional Similarity Consistency (RSC) constraint between these two modalities to suppress the disturbance of background in RGB features. Specifically, we calculate the regional similarity matrices of RGB and Flow features, and impose the consistency constraint through $L_2$ loss. To verify the effectiveness of our method, we integrate the proposed RSC constraint into three recent methods. The comprehensive experimental results show that the proposed RSC constraint can boost the performance of these methods, and achieve the state-of-the-art results on the widely-used THUMOS14 and ActivityNet1.2 datasets.

Authors: Yanrui Niu, Jingyao Yang, Chao Liang, Baojin Huang, Zhongyuan Wang

Abstract:

As one of the challenging problems in video search, PersonAction Instance Search (P-A INS) aims to retrieve shots with a specific person carrying out a specific action from massive video shots. Most existing methods conduct person INS and action INS separately to compute the initial person and action ranking scores, which will be directly fused to generate the final ranking list. However, direct aggregation of two individual INS scores ignores spatial relationships of person and action, thus cannot guarantee their identity consistency and cause identity inconsistency problem (IIP). To address IIP, we propose a simple spatio-temporal identity verification method. Specifically, in the spatial dimension, we propose an identity consistency verification (ICV) step to revise the direct fusion score of person INS and action INS. Moreover, in the temporal dimension, we propose a double-temporal extension (DTE) operation, i.e., intra-shot DTE and inter-shot DTE. The former improves the coverage of ICV by adding detection results on keyframes, and the latter improves the accuracy of P-A INS by using the coherence of action among shots in movies. The proposed method is evaluated on the large-scale NIST TRECVID INS 2019-2021 tasks, and the experimental results show that it can effectively mitigate the IIP, and its performance surpasses that of the champion team in 2019 INS task and the second place teams in both 2020 and 2021 INS tasks.

Authors: Hongfeng Han, Zhiwu Lu, Ji-Rong Wen

Abstract:

In the typical video action classification scenario, it is critical to extract the temporal and spatial information in the videos with complex 3D convolution neural networks, which significantly improves the computation cost and memory costs. In this paper, based on the binary neural network, we propose a novel 1-bit 3D convolution block named 3D BitConv, capable of compressing the 3D convolutional networks while maintaining high accuracy. Due to its high flexibility, the proposed 3D BitConv block can be directly embedded in the latest 3D convolutional backbone. Consequently, we binarize two representative 3D convolutional neural networks (C3D and ResNet3D) and evaluate their performance on action recognition tasks. Extensive experiments on two widely used action recognition datasets demonstrate that the two binary 3D networks achieve impressive performance at a lower cost. Furthermore, we conduct a comprehensive ablation study to test and verify the effectiveness of the components in the proposed 3D BitConv.

Authors: Amato, Giuseppe; Bolettieri, Paolo; Carrara, Fabio; Falchi, Fabrizio; Gennaro, Claudio; Messina, Nicola; Vadicamo, Lucia; Vairo, Claudio

Abstract:

In this paper, we present the fourth release of VISIONE, a tool for fast and effective video search on a large-scale dataset. It includes several search functionalities like text search, object and color-based search, semantic and visual similarity search, and temporal search. VISIONE uses ad-hoc textual encoding for indexing and searching video content, and it exploits a full-text search engine as search backend.

Authors: Spiess, Florian; Heller, Silvan; Rossetto, Luca; Sauter, Loris; Weber, Philipp; Schuldt, Heiko

Abstract:

Virtual reality (VR) interfaces allow for entirely new modes of user interaction with systems and interfaces. Much like in physical workspaces, documents, tools, and interfaces can be used, put aside, and used again later. Such asynchronous workflows are a great advantage of virtual environments, as they enable users to perform multiple tasks in an interleaved manner, allowing partial results or insights from one task to affect the choices made in another. However, the usage patterns of such workflows are much more challenging to analyze. In this paper we present the version of vitrivr-VR participating in the Video Browser Showdown (VBS) 2023. We describe the current state of our system, with a focus on text input methods and logging of asynchronous workflows.

Authors: Lokoč, Jakub; Vopálková, Zuzana; Dokoupil, Patrik; Peška, Ladislav

Abstract:

Nowadays, deep learning based models like CLIP allow simple design of cross-modal video search systems able to solve many tasks considered as highly challenging several years ago. In this paper, we analyze a CLIP based search approach that focuses on situations, where users cannot find proper text queries to describe searched video segments. The approach relies on suggestions of classes for displayed intermediate result sets and thus allows users to realize missing words and ideas to describe video frames. This approach is supported with a preliminary study showing potential of the method. Based on the results, we extend a respected known-item search system for the Video Browser Showdown, where more challenging visual known-item search tasks are planned.

Authors: Lubos, Sebastian; Rubino, Massimiliano; Tautschnig, Christian; Tautschnig, Markus; Wen, Boda; Schoeffmann, Klaus; Felfernig, Alexander

Abstract:

This paper presents the first version of our video search application Perfect Match for the Video Browser Showdown 2023 competition. The system indexes videos from the large V3C video dataset and derives visual content descriptors automatically. Further, it provides an interactive video search UI, which implements approaches from the domain of Critiquing-based recommendation.

Authors: Song, Weixi; He, Jiangshan; Li, Xinghan; Feng, Shiwei; Liang, Chao

Abstract:

In this paper, we present a quantum-inspired interactive video search engine (QIVISE), which will be tested in VBS2023. QIVISE aims at assisting the user in dealing with Known-Item Search and Ad- hoc Video Search tasks with high efficiency and accuracy. QIVISE is based on a text-image encoder to achieve multi-modal embedding and introduces multiple interaction possibilities, including a novel quantum- inspired interaction on paradigm, label search, and multi-modal search to refine the retrieval results via user’s interaction and feedback.

Authors: Sauter, Loris; Gasser, Ralph; Heller, Silvan; Rossetto, Luca; Saladin, Colin; Spiess, Florian; Schuldt, Heiko

Abstract:

vitrivr is a general purpose retrieval system that supports a wide range of query modalities. In this paper, we briefly introduce the system and describe the changes and adjustments made for the 2023 iteration of the video browser showdown. These focus primarily on text-based retrieval schemes and corresponding user-feedback mechanisms.

Authors: Hoang-Xuan, Nhat; Nguyen, E-Ro; Nguyen-Ho, Thang-Long; Pham, Minh-Khoi; Nguyen, Quang-Thuc; Trang-Trung, Hoang-Phuc; Ninh, Van-Tu; Le, Tu-Khiem; Gurrin, Cathal; Tran, Minh-Triet

Abstract:

In this paper, we present a new version of our interactive video retrieval system V-FIRST. Besides the existing features of querying by textual descriptions and visual examples, we include a novel referring expression segmentation module to highlight the objects in an image. This is the first step towards providing adequate explainability to retrieval results, ensuring that the system can be trusted and used in domain-specific and critical scenarios. Searching by sequence of events is also a new addition, as it proves to be pivotal in finding events from memory. Furthermore, we improved our Optical Character Recognition capability, especially in the case of scene text. Finally, the inclusion of relevance feedback allows the user to explicitly refine the search space. All combined, our system has greatly improved for user interaction, leveraging more explicit information and providing more tools for the user to work with.

Authors: Pantelidis, Nick; Andreadis, Stelios; Pegia, Maria; Moumtzidou, Anastasia; Galanopoulos, Damianos; Apostolidis, Konstantinos; Touska, Despoina; Gkountakos, Konstantinos; Gialampoukidis, Ilias; Vrochidis, Stefanos; Mezaris, Vasileios; Kompatsiaris, Ioannis

Abstract:

This paper describes VERGE, an interactive video retrieval system for browsing a collection of images from videos and searching for specific content. The system utilizes many retrieval techniques as well as fusion and reranking capabilities. A Web Application is also part of VERGE, where a user can create queries, view the top results and submit the appropriate data, all in a user-friendly way.

Authors: Schall, Konstantin; Hezel, Nico; Barthel, Kai Uwe; Jung, Klaus

Abstract:

Vibro represents a powerful tool for interactive video retrieval and browsing and is the winner of the Video Browser Showdown 2022. Following the saying of "never change a winning system" we did not change any of the underlying concepts or added any new features. Instead, we focused on improving the three existing corner stones of the software, which are text-to-image search, image-to-image retrieval and browsing of results with 2D sorted maps. The changes to these three parts are summarized in this paper and additionally, a high-level overview of the AVS-mode of vibro is presented.

Authors: Nguyen, Thao-Nhu; Puangthamawathanakun, Bunyarit; Gurrin, Cathal; Caputo, Annalina; Healy, Graham; Nguyen, Binh T.; Arpnikanondt, Chonlameth

Abstract:

In this paper, we present an interactive video retrieval system named VideoCLIP developed for the Video Browser Showdown 2023. To support users in solving retrieval tasks, the system enables search using a variety of modalities, such as rich text, dominant colour, OCR, and query-by-image. Moreover, a new search modality has been added to empower our core engine, which is inherited from the Contrastive Language–Image Pre-training (CLIP) model. Finally, the user interface is enhanced to display results in groups in order to reduce the effort for a user when locating potentially relevant targets.

Authors: Arnold, Rahel; Sauter, Loris; Schuldt, Heiko

Abstract:

Due to the ever increasing amount of multimedia data, efficient means for multimedia management and retrieval are required. Especially with the rise of deep-learning-based analytics methods, the semantic gap has shrunk considerably, but a human in the loop is still considered mandatory. One of the driving factors of video search is that humans tend to refine their queries after reviewing the results. Hence, the entire process is highly interactive. A natural approach to interactive video search is using textual descriptions of the content of the expected result, enabled by deep learning-based joint visual text co-embedding. In this paper, we present the Multi-Modal Multimedia Retrieval (4MR) system, a novel system inspired by vitrivr, that empowers users with almost entirely free-form query formulation methods. The top-ranked teams of the last few iterations of the Video Browser Showdown have shown that CLIP provides an ideal feature extraction method. Therefore, while 4MR is capable of image and text retrieval as well, for VBS video retrieval is based primarily based on CLIP.

Authors: Schoeffmann, Klaus; Stefanics, Daniela; Leibetseder, Andreas

Abstract:

The diveXplore system has been participating in the VBS since 2017 and uses a sophisticated content analysis stack as well as an advanced interface for concept, object, event, and texts search. This year, we perform several changes in order to make the system both much easier to use as well as more efficient. These changes include using shot-based analysis over uniform sampled segments, integration of textual and image transformers (CLIP), as well as optimizing the user interface for faster browsing of video summaries.

Authors: MA, Zhixin; WU, Jiaxin; LOO, Weixiong; NGO, Chong wah

Abstract:

With the tremendous increase of video data size, the search performance could be impacted significantly. Specifically, in an interactive system, a real-time system allows a user to quickly browse, search and refine a query. Without a speedy system, which is the main ingredient to engage a user to stay focus, interactive system becomes less effective even with a sophisticated deep learning system. This paper addresses this challenge by leveraging approximate search, Bayesian inference and reinforcement learning. For approximate search, we apply hierarchical navigable small world, which is an efficient approximate nearest neighbor search algorithm. To quickly prune the search scope, we integrate PicHunter, one of the most popular engines in Video Browser Showdown, with reinforcement learning. The integration enhances PicHunter with the ability of systematic planning. Specifically, PicHunter performs Bayesian update with greedy strategy to select a small number of candidates for display. With reinforcement learning, the greedy strategy is replaced with a policy network that learns to select candidates that will result in minimum number of user iterations, which is analytically defined by a reward function. With these improvements, the interactive system only searches a subset of video dataset relevant to a query while being able to quickly perform Bayesian update with systematic planning to recommend the most probable candidates that can potentially lead to minimum iteration rounds.

Authors: Hao, Yuzhe; Uto, Kuniaki; Kanezaki, Asako; Sato, Ikuro; Kawakami, Rei; Shinoda, Koichi

Abstract:

Egocentric Human Activity Recognition (ego-HAR) has received attention in fields where human intentions in a video must be estimated. The performance of existing methods, however, are limited due to insufficient information about the subject's motion in egocentric videos. We consider that a dataset of egocentric videos along with two inertial sensors attached to both wrists of the subject to obtain more information about the subject's motion will be useful to study the problem in depth. Therefore, this paper provides a publicly available dataset, EvIs-Kitchen, which contains well-synchronized egocentric videos and two-hand inertial sensor data, as well as interaction-highlighted annotations. We also present a baseline multimodal activity recognition method with two-stream architecture and score fusion to validate that such multimodal learning on egocentric videos and intertial sensor data is more effective to tackle the problem. Experiments show that our multimodal method outperforms other single-modal methods on EvIs-Kitchen.

Authors: Zhou, Longlong; Wu, Xiao-Jun; Xu, Tianyang

Abstract:

Language-based image generation is a challenging task. Current studies normally employ conditional generative adversarial network (cGAN) as the model framework and have achieved significant progress. Nonetheless, a close examination of their methods reveals two fundamental issues. First, the discrete linguistic conditions make the training of cGAN extremely difficult and impair the generalization performance of cGAN. Second, the conditional discriminator cannot extract semantically consistent features based on linguistic conditions, which is not conducive to conditional discrimination. To address these issues, we propose a condition optimization and mutual information maximization GAN (COMIM-GAN). To be specific, we design (1) a text condition construction module, which can construct a compact linguistic condition space, and (2) a mutual information loss between images and linguistic conditions to motivate the discriminator to extract more features associated with the linguistic conditions. Extensive experiments on CUB-200 and MS-COCO datasets demonstrate that our method is superior to the existing methods.

Authors: Lokoc, Jakub; Peska, Ladislav

Abstract:

Recently, the CLIP model demonstrated impressive performance in text-image search and zero classification tasks. Hence, CLIP was used as a primary model in many cross-modal search tools at evaluation campaigns. In this paper, we show a study performed with the model integrated to a successful video search tool at respected Video Browser Showdown competition. The tool allows more complex querying actions on top of the primary model. Specifically, temporal querying and Bayesian like relevance feedback were tested as well as their natural combination -- temporal relevance feedback. In a thorough analysis of the tool's performance, we show current limits of cross-modal searching with CLIP and also the impact of more advanced query formulation strategies. We conclude that current cross-modal search models enable users to solve some types of tasks trivially with a single query, however, for more challenging tasks it is necessary to rely also on interactive search strategies.

Authors: Pham, Dinh-Duy; Dao, Minh-Son; Nguyen, Thanh-Binh

Abstract:

Dashcam video has become popular recently due to the safety of both individuals and communities. While individuals can have undeniable evidence for legal and insurance, communities can benefit from sharing these dashcam videos for further traffic education and criminal investigation. Moreover, relying on recent computer vision and AI development, a few companies have launched the so-called AI dashcam that can alert drivers to near-risk accidents (e.g., following distance detection, forward collision warning) to improve driver safety. Unfortunately, even though dashcam videos create a driver's travel log (i.e., a traveling diary), little research focuses on creating a valuable and friendly tool to find any incident or event with few described sketches by users. Inspired by these observations, we introduce an interactive incident detection and retrieval system for first-view travel-log data that can retrieve fine-grained incidents for both defined and undefined incidents. Moreover, the system gives promising results when evaluated on several public datasets and popular text-image retrieval methods. The source code is published at \textit{https://github.com/PDD0911-HCMUS/Cross\_Model\_Attention}

Authors: Liang, Mingliang; Liu, Zhuoran; Larson, Martha

Abstract:

We propose a Textual Concept Expansion (TCE) approach for creating joint textual-visual embeddings. TCE uses a multi-label classifier that takes a caption as input and produces as output a set of concepts that are used to expand, i.e., enrich the caption. TCE addresses the challenge of the limited number of concepts common between an image and its caption by leveraging general knowledge about the world, i.e., commonsense knowledge. Following a recent trend, the commonsense knowledge is acquired by creative use of the training data. We test TCE within a popular dual-stream approach, Consensus-Aware Visual-Semantic Embedding (CVSE). This popular approach leverages a graph that encodes the co-occurrence of concepts, which it takes to represent a consensus between the textual and visual modality that captures commonsense knowledge. Experimental results demonstrate an improvement of image-text matching when TCE is used for the expansion of the background collection and the query. Query expansion, not possible in the original CVSE, is particularly helpful. TCE can be extended in the future to make use of data that is similar to the target domain, but is drawn from an additional, external data set.

Authors: Nik, Alireza Hossein Zadeh; Riegler, Michael A.; Halvorsen, Pål; Storås, Andrea M.

Abstract:

High-quality tabular data is a crucial requirement for developing data-driven applications, especially healthcare-related ones, because most of the data nowadays collected in this context is in tabular form. However, strict data protection laws complicates the access to medical datasets. Thus, synthetic data has become an ideal alternative for data scientists and healthcare professionals to circumvent such hurdles. Although many healthcare institutions still use the classical de-identification and anonymization techniques for generating synthetic data, deep learning-based generative models such as Generative Adversarial Networks (GANs) have shown a remarkable performance in generating tabular datasets with complex structures. This paper examines the GANs’ potential and applicability within the healthcare industry, which often faces serious challenges with insufficient training data and patient records sensitivity. We investigate several state-of-the-art GAN-based models proposed for tabular synthetic data generation. Healthcare datasets with different sizes, numbers of variables, column data types, feature distributions, and inter-variable correlations are examined. Moreover, a comprehensive evaluation framework is defined to evaluate the quality of the synthetic records and the viability of each model in preserving the patients’ privacy. The results indicate that the proposed models can generate synthetic datasets that maintain the statistical characteristics, model compatibility and privacy of the original data. Moreover, synthetic tabular healthcare datasets can be a viable option in many data-driven applications. However, there is still room for further improvements in designing a perfect architecture for generating synthetic tabular data.

Authors: Le, Quoc-Cuong; Le, Minh-Quan; Tran, Mai-Khiem; Le, Ngoc-Quyen; Tran, Minh-Triet

Abstract:

Flooding in urban areas is one of the serious problems and needs special attention in urban development and improving people's living quality. Flood detection to promptly provide data for hydrometeorological forecasting systems will help make timely forecasts for life. In addition, providing information about rain and flooding in many locations in the city will help people make appropriate decisions about traffic. Therefore, in this paper, we present our FL-Former solution for detecting and classifying rain and inundation levels in urban locations, specifically in Ho Chi Minh City, based on images recorded from cameras using Vision Transformer. We also build the HCMC-URF dataset with more than 10K images of various rainy and flooding conditions in Ho Chi Minh City to serve the community's research. Finally, we propose the software architecture and construction of an online API system to provide timely information about raining and flooding at several locations in the city as extra input for hydrometeorological analysis and prediction systems, as well as provide information to citizens via mobile or web applications.

Authors: Chen, Shuo; Li, Di; Ju, Bobo; Jiang, Linhua; Zhao, Dongfang

Abstract:

The texture scarcity properties make transparent object localization a challenging task in the computer vision community. This paper addresses this task in two aspects. (i) Additional guidance cues: we propose a Simulation Heatmap Guidance (SHG) to improve the localization ability of the model. Concretely, the target’s extreme points and inference centroids are used to generate simulation heatmaps to offer additional position guides. A high recall is rewarded even in extreme cases. (ii) Enhanced attention: we propose a Context Spatial Attention (CSA) combined with a unique backbone to build dependencies between feature points and to boost multi-scale attention fusion. CSA is a lightweight module and brings apparent perceptual gain. Experiments show that our method achieves more accurate detection for cluttered transparent objects in various scenarios and background settings, outperforming the existing methods.

Authors: Ye, Kai; Ji, Haoqin; Li, Yuan; Wang, Lei; Liu, Peng; Shen, Linlin

Abstract:

Human parts detection has made remarkable progress due to the development of deep convolutional networks. However, many SOTA detection methods require large computational cost and are still difficult to be deployed to edge devices with limited computing resources. In this paper, we propose a lightweight Cascade Center-based Framework, called CCF-Net, for human parts detection. Firstly, a Gaussian-Induced penalty strategy is designed to ensure that the network can handle objects of various scales. Then, we use Cascade AttentionModule to capture relations between different feature maps, which refines intermediate features. With our novel cross-dataset training strategy, our framework fully explores the datasets with incomplete annotations and achieves better performance. Furthermore, Center-based Knowledge Distillation is proposed to enable student models to learn better representation without additional cost. Experiments show that our method achieves a new SOTA performance on Human-Parts and COCO Human Parts benchmarks.

Authors: Dong, Ke; Peng, Hao; Che, Jie

Abstract:

The dynamic-static fusion features play an important role in speech emotion recognition (SER). However, the fusion methods of dynamic features and static features generally are simple addition or serial fusion, which might cause the loss of certain underlying emotional information. To address this issue, we proposed a dynamic-static cross attentional feature fusion method (SD-CAFF) with a cross attentional feature fusion mechanism (Cross AFF) to extract superior deep dynamic-static fusion features. To be specific, the Cross AFF is utilized to parallel fuse the deep features from the CNN/LSTM feature extraction module, which can extract the deep static features and the deep dynamic features from acoustic features (MFCC, Delta, and Delta-delta). In addition to the SD-CAFF framework, we also employed muti-label auxiliary learning in the training process to further improve the accuracy of emotion recognition. The experimental results on IEMOCAP demonstrated the WA and UA of SD-CAFF are 75.78% and 74.89%, respectively, which outperformed the current SOTAs. Furthermore, SD-CAFF achieved competitive performances (WA: 56.77%; UA: 56.30%) in the comparison experiments of cross-corpus capability on MSP-IMPROV. Finally, we ablation experiments to verify the effectiveness of SD-CAFF and the necessity of each module.

Authors: John, Vijay Cornelius Kirubakaran; Kawanishi, Yasutomo

Abstract:

Audio-visual person recognition is the problem of recognizing a person’s identity from the multimodal audio-visual data. Audio-visual person recognition has many applications in security, surveillance, biometrics etc. Deep learning-based audio-visual camera person recognition report state-of-the-art person recognition accuracy. However, existing audio-visual frameworks require the presence of both modalities. This approach is limited by the problem of missing modalities, where one or more of the modalities could be missing. In this paper, we formulate an audio-visual person recognition framework where we define and address the missing visual modality problem. The proposed framework enhances the robustness of audio-visual person recognition even under the condition of missing visual modality using audio-based person attributes and a multi-head attention transformer-based network, termed the CTNet. The audio-based person attributes such as age, gender and race are predicted from the audio data using a deep learning model, termed the S2A network. The attributes predicted from the audio data, which are assumed to be always available, provide additional cues for the person recognition framework. The predicted attributes, the audio data and the image data, which may be missing, are given as input to the CTNet, which contains the multi-head attention branch. The multi-head attention branch addresses the problem of missing visual modality by assigning attention weights to the audio features, visual features and the audio-based attributes. The proposed framework is validated with the CREMA-D public dataset using comparative analysis and an ablation study. The results show that the proposed framework enhances the robustness of person recognition even under the condition of missing visible camera.

Authors: Chen, Tianrun; Fu, Chenglong; Zang, Ying; Zhu, Lanyun; Zhang, Jia; Mao, Papa; Sun, Lingyun

Abstract:

The rapid development of AR/VR brings tremendous demands for 3D content. While the widely-used Computer-Aided Design (CAD) method requires a time-consuming and labor-intensive modeling process, sketch-based 3D modeling offers a potential solution as a natural form of computer-human interaction. However, the sparsity and ambiguity of sketches make it challenging to generate high-fidelity content reflecting creators' ideas. Precise drawing from multiple views or strategic step-by-step drawings is often required to tackle the challenge but is not friendly to novice users. In this work, we introduce a novel end-to-end approach, Deep3DSketch+, which performs 3D modeling using only a single free-hand sketch without inputting multiple sketches or view information. Specifically, we introduce a lightweight generation network for efficient inference in real-time and a structural-aware adversarial training approach with a Stroke Enhancement Module (SEM) to capture the structural information to facilitate learning of the realistic and fine-detailed shape structures for high-fidelity performance. Extensive experiments demonstrated the effectiveness of our approach with the state-of-the-art (SOTA) performance on both synthetic and real datasets.

Authors: Li, Yuhang; Cai, Feifan; Tu, Yifei; Ding, Youdong

Abstract:

The low visibility of low-light images due to lack of exposure poses a significant challenge for vision tasks such as image fusion, detection and segmentation in low-light conditions. Real-world situations such as backlighting and shadow occlusion mostly exist with non-uniform low-light, while existing enhancement methods tend to brighten both low-light and normal-light regions, we actually prefer to enhance dark regions but suppress overexposed regions. To address this problem, we propose a new non-uniform dark visual network (NDVN) that uses the attention mechanism to enhance regions with different levels of illumination separately. Since deep-learning needs strong data-driven, for this purpose we carefully construct a non-uniform dark synthetic dataset (UDL) that is larger and more diverse than existing datasets, and more importantly it contains more non-uniform light states. We use the manually annotated luminance domain mask (LD-mask) in the dataset to drive the network to distinguish between low-light and extremely dark regions in the image. Guided by the LD-mask and the attention mechanism, the NDVN adaptively illuminates different light regions while enhancing the color and contrast of the image. More importantly, we introduce a new region loss function to constrain the network, resulting in better quality enhancement results. Extensive experiments show that our proposed network outperforms other state-of-the-art methods both qualitatively and quantitatively.

Authors: Dong, Aimei; Liu, Sidi

Abstract:

Recently, single-task learning on semantic segmentation tasks has achieved good results. When multiple tasks are handled simultaneously, single-task learning requires an independent network structure for each task and no intersection between tasks. This paper proposes a feature fusion-attention mechanism multi-task learning method, which can simultaneously handle multiple related tasks (semantic segmentation, surface normal estimation task, etc.). Our model includes a feature extraction module to extract semantic information at different scales, a feature fusion module to refine the extracted features, and an attentional mechanism for processing information from fusion modules to learn information about specific tasks. The network architecture proposed in this paper trains in an end-to-end manner and simultaneously improves the performance of multiple tasks. Experiments are carried out on two well-known semantic segmentation datasets, and the accuracy of the proposed model is verified.

Authors: Zhang, Xinxin; Pan, Shanliang; Qian, Chengwu; Yuan, Jiadong

Abstract:

With the rapid development of the Internet, social media has become the main platform for users to obtain news and share their opinions. While social media provides convenience to the life of people, it also offers advantageous conditions for publishing and spreading rumors. The rumors widely propagated on social media can cause mass panic and damage personal reputation. Since artificial de-tection methods take a lot of time, it becomes crucial to use intelligent methods for rumor detection. The recent rumor detection methods mostly use the meta-paths of post propagation to construct isomorphic graphs to find clues in the propaga-tion structure. However, these methods do not fully use the global and local rela-tions in the propagation graph and do not consider the correlation between differ-ent types of nodes. In this paper, we propose a Global-Local Relations Encoding Network (GLREN), which encodes node relations in the heterogeneous graph from global and local perspectives. First, we explore the semantic similarity be-tween all source posts and comment posts to generate global and local semantic representations. Then, we construct user credibility levels and interaction relations to explore the potential relationship between users and misinformation. Finally, we introduce a root enhancement strategy to enhance the influence of source posts and publisher information. The experimental results show that our model can outperform the accuracy of the state-of-the-art methods by 3.0% and 6.0% on Twitter15 and Twitter16, respectively.

Authors: Yang, Yi-Ting; Chu, Wei-Ta

Abstract:

We especially target at detecting text in atypical font styles and in cluttered background for Japanese comics (manga). To enable the detection model to detect atypical text, we augment training data by the proposed manga-specific data augmentation. A generative adversarial network is developed to generate atypical text regions, which are then blended into manga pages to largely increase the volume and diversity of training data. We verify the importance of manga-specific data augmentation. Furthermore, with the help of manga text detection, we fuse global visual features and local text features to enable more accurate emotion analysis.

Authors: Gong, Fucai; Xie, Yuchen; Jiang, Le; Chen, Keming (1); Liu, Yunxin; Ye, Xiaozhou; Ouyang, Ye

Abstract:

Few-shot learning is increasingly popular in image classification. The key is to learn the significant features from source classes to match the support and query pairs. In this paper, we redesign the contrastive learning scheme in a few-shot manner with selected proposal boxes generated by Navigator network. The main work of this paper includes: (i) We analyze the limitation of hard sample generating proposed by current few-shot learning methods with contrastive learning and find additional noise introduced in contrastive loss construction. (ii) We propose a novel embedding model with contrastive learning named infoPB which improves hard samples with proposal boxes to improve Noise Contrastive Estimation. (iii) We demonstrate infoPB is effective in few-shot image classification and benefited from Navigator network through the ablation study. (iv) The performance of our method is evaluated thoroughly on typical few-shot image classification tasks. It verifies a new state-of-the-art performance compared with outstanding competitors with their best results on miniImageNet in 5-way, 5-shot, and tieredImageNet in 5-way, 1-shot/5-way, 5-shot.

Authors: Zheng, Minyan; Luo, Jianping

Abstract:

Space-time video super-resolution, which aims to generate a high resolution (HR) and high frame rate (HRF) video from a low frame rate (LFR), low resolution (LR) video. Simply combining video frame interpolation (VFI) and video super-resolution (VSR) network to solve this problem cannot bring satisfying performance, which also requires a heavy computational burden. In this paper, we investigate a one-stage network to jointly up-sample video both in time and space. In our framework, a 3D pyramid structure with channel attention is proposed to fuse input frames and generate intermediate features. The features are fed into the 3D Transformer network to model global relationships between features. Our proposed network, 3DTFSR, can efficiently process videos without explicit motion compensation. Extensive experiments on benchmark datasets demonstrate that the proposed method achieves better quantitative and qualitative performance compared to a two-stage network.

Authors: Wu, Jinmeng; Shu, Pengcheng; Hong, Hanyu; Li, Xingxun; Ma, Lei; Zhang, Yaozong; Zhu, Ying; Wang, Lei

Abstract:

For the anomaly detection task of video sequences, CNN-based methods have been able to learn to describe the normal situation without abnormal samples at training time by reconstructing the input frame or predicting the future frame, and then use the reconstruction error to represent the abnormal situation at testing time. Transformers, however, have achieved the same spectacular outcomes as CNN on many tasks after being utilized in the field of vision, and they have also been used in the task of anomaly detection. We present an unsupervised learning method based on Vision Transformer in this work. The model has an encoder-decoder structure, and the Memory module is used to extract and en-hance the video sequence's local pattern features. We discover anomalies in various data sets and visually compare distinct scenes in the data set. The experimental results suggest that the model has a significant impact on the task of dealing with anomalies.

Authors: Zhong, Shanshan; Wen, Wushao; Qin, Jinghui

Abstract:

Recently many effective attention modules are proposed to boot the model performance by exploiting the internal information of convolutional neural networks in computer vision. In general, many previous works ignore considering the design of the pooling strategy of the attention mechanism since they adopt the global average pooling for granted, which hinders the further improvement of the performance of the attention mechanism. However, we empirically find and verify a phenomenon that the simple linear combination of global max-pooling and global min-pooling can produce pooling strategies that match or exceed the performance of global average pooling. Based on this empirical observation, we propose a simple-yet-effective attention module SPEM, which adopts a self-adaptive pooling strategy based on global max-pooling and global min-pooling and a lightweight module for producing the attention map. The effectiveness of SPEM is demonstrated by extensive experiments on widely-used benchmark datasets and popular attention networks.

Authors: Xu, Haoqi; Hou, Jian; Yuan, Huaqiang

Abstract:

As real-world data are often represented by multiple sets of features in different views, it is desirable to improve clustering results with respect to ordinary single-view clustering by making use of the consensus and complementarity among different views. For this purpose, weighted multi-view clustering is proposed to combine multiple individual views into one single combined view, which is used to generate the final clustering result. In this paper we present a simple yet effective weighted multi-view clustering algorithm based on internal evaluation of clustering results. Observing that an internal evaluation criterion can be used to estimate the quality of clustering results, we propose to weight different views to maximize the clustering quality in the combined view. We firstly introduce an implementation of the Dunn index and a heuristic method to determine the scale parameter in spectral clustering. Then an adaptive weight initialization and updating method is proposed to improve the clustering results iteratively. Finally we do spectral clustering in the combined view to generate the clustering result. In experiments with several publicly available image and text datasets, our algorithm compares favorably or comparably with some other algorithms.

Authors: Touska, Despoina; Gkountakos, Konstantinos; Ioannidis, Konstantinos; Tsikrika, Theodora; Vrochidis, Stefanos; Kompatsiaris, Ioannis

Abstract:

In Multiple Object Tracking (MOT), data association is a key component of the tracking-by-detection paradigm and endeavors to link a set of discrete object observations across a video sequence, yielding possible trajectories. Our intention is to provide a classification of numerous graph-based works according to the way they measure object dependencies and their footprint on the graph structure they construct. In particular, methods are organized into Measurement-to-Measurement (MtM), Measurement-to-Track (MtT), and Track-to-Track (TtT). At the same time, we include recent Deep Learning (DL) implementations among traditional approaches to present the latest trends and developments in the field and offer a performance comparison. In doing so, this work serves as a foundation for future research by providing newcomers with information about the graph-based bibliography of MOT.

Authors: Han, Hongfeng; Lu, Zhiwu; Wen, Ji-Rong

Abstract:

In video action segmentation scenarios, sufficient training data is needed for training a deep learning model. However, due to the high cost of human annotation for action segmentation, only very limited training videos can be accessible. Further, large spatio-temporal variations exist between the training and test data. Therefore, it is critical to obtain effective representations with limited training videos and efficiently utilize unlabeled test videos. To this end, based on self-supervised learning and domain adaptation, we propose a novel Contrastive Temporal Domain Adaptation (CTDA) framework for action segmentation. Specifically, in the self-supervised learning module, two auxiliary tasks are defined based on binary and sequential domain prediction and then are addressed by the combination of domain adaptation and contrastive learning. Further, a multi-stage architecture is devised with one prediction generation stage and three refinement stages to obtain the final results of action segmentation. Extensive experiments on two action segmentation benchmarks demonstrate that the proposed CTDA achieves state-of-the-art performance.

Authors: Veselý, Patrik; Peska, Ladislav

Abstract:

The concept of object-to-object similarity plays a crucial role in interactive content-based video retrieval tools. Similarity (or distance) models are core components of several retrieval concepts, e.g. Query by Example or relevance feedback. In these scenarios, the common approach is to apply some feature extractor that transforms the object to a vector of features, i.e., positions it into an induced latent space. The similarity is then based on some distance metric in this space. Historically, feature extractors were mostly based on some color histograms or hand-crafted descriptors such as SIFT, but nowadays state-of-the-art tools mostly rely on some deep learning (DL) approaches. However, so far there were no systematic study of how suitable are individual feature extractors in the video retrieval domain. Or, in other words, to what extent are human-perceived and model-based similarities concordant. To fill this gap, we conducted a user study with over 4000 similarity judgements comparing over 20 variants of feature extractors. Results corroborate the dominance of deep learning approaches, but surprisingly favor smaller and simpler DL models instead of larger ones.

Authors: yan, zhiqi; liang, shuang

Abstract:

Although deep convolutional networks have achieved good results in the field of salient object detection, most of these methods cannot work well near the boundary. This results in poor boundary quality of network predictions, accompanied by a large number of blurred contours and hollow objects. To solve this problem, this paper proposes a Boundary Enhance Network (BENet) for salient object detection, which makes the network pay more attention to salient edge features by fusing auxiliary boundary information of objects. We adopt the Progressive Feature Extraction Module (PFEM) to obtain multi-scale edge and object features of saliency objects. In response to the semantic gap problem in feature fusion, we propose an Adaptive Edge Fusion Module (AEFM) to allow the network to adaptively and complementarily fuse edge features and salient object features. The Self Refinement (SR) module further repairs and enhances edge features. Moreover, in order to make the network pay more attention to the boundary, we design an edge enhance loss function, which uses the additional boundary maps to guide the network to learn rich boundary features at the pixel level. Experimental results show that our proposed method outperforms state-of-the-art methods on five benchmark datasets.

Authors: Niu, Chaoqun; Li, Yuan; Wang, Jian (1); Zhou, Jizhe; Xiong, Tu; Yu, Dong; Guo, Huili; Zhang, Lin; Liang, Weibo; Lv, Jiancheng

Abstract:

Activating bone from a chest X-ray (CXR) is significant for disease diagnosis and health equity for under-developed areas, while the complex overlap of anatomical structures in CXR constantly challenges bone activation performance and adaptability. Besides, due to high data collection and annotation costs, no large-scale labeled datasets are available. As a result, existing methods commonly use single-view CXR with annotations to activate bone. To address these challenges, in this paper, we propose an adaptive bone activation framework. This framework leverages the Dual-Energy Subtraction (DES) images to consist of multi-view image pairs of the CXR and the contrastive learning theory to construct training samples. In particular, we first devise a Siamese/Triplet architecture supervisor; correspondingly, we establish a cGAN-styled activator based on the learned skeletal information to generate the bone image from the CXR. To our knowledge, the proposed method is the first multi-view bone activation framework obtained without manual annotation and has more robust adaptability. The mean of Relative Mean Absolute Error and the Fréchet Inception Distance are 3.45% and 1.12 respectively, which proves the results activated by our method retain more skeletal details with few feature distribution changes. From the visualized results, our method can activate bone images from a single CXR ignoring overlapping areas. Bone activation has drastically improved compared to the original images.

Authors: Yan, Zhaoyong; Ma, Liyan; Luo, Xiangfeng; Sun, Yan

Abstract:

The goal of rain streak removal is to recover the rain-free background scenes of an image degraded by rain streaks. Most current deep convolutional neural networks methods have achieved dramatic performance. However, these methods still cannot capture the discriminative features to well distinguish the rain streaks and the important image content. To solve this problem, we propose a Multi-scale and Multi-stage deraining network in the end-to-end manner. Specifically, we design a multi-scale rain streak extraction module to capture complex rain streak features across different scales through the multi-scale selection kernel attention mechanism. In addition, multi-stage learning is used to extract deeper feature representations of rain streak and fuse different stages of background information. Furthermore, we introduce a Fourier space loss function to reduce the loss of high-frequency information in the background image and improve the quality of deraining results. Extensive experiments demonstrate that our network performs favorably against the state-of-the-art deraining methods.

Authors: Wang, Wanliang; Xing, Fangsen; Chen, Jiacheng; Tu, Hangyao

Abstract:

High-resolution magnetic resonance (MR) imaging is beneficial for accurate disease diagnosis and subsequent analysis. Currently, the single image super-resolution (SR) technique is an effective and less costly alternative technique to improve the spatial resolution of MR images. Structural information in MR images is crucial during clinical diagnosis, but it is often ignored by existing deep learning MR image SR technique. Consequently, we propose edge assisted feature extraction block (EAFEB), which can efficiently extract the content and edge features from low-resolution (LR) images, allowing the network to focus on both content and geometric structure. To fully utilize the features extracted by EAFEB, an asymmetric convolutional group (ACG) is proposed, which can balance structural feature preservation and content feature extraction. Moreover, we design a novel contextual spatial attention (CSA) method to facilitate the network focus on critical information. Experiment results in various MR image sequences, including T1, T2, and PD, show that our Edge Assisted Asymmetric Convolution Network (EAACN) has superior results relative to recent leading SR models.

Authors: Nguyen Mau, Trong-Hieu; Trinh, Quoc-Huy; Bui, Nhat-Tan; Vo Thi, Phuoc-Thao; Nguyen, Minh-Van; Cao, Xuan-Nam; Tran, Minh-Triet; Nguyen, Hai-Dang

Abstract:

With the development of biomedical computing, the segmentation task has an integral part in helping the doctor correctly identify the position of the polyps or the ache in the system. However, precise polyp segmentation is challenging because the same type of polyps has a diversity of size, color, and texture; previous methods cannot fully transfer information from encoder to decoder due to the lack of details and knowledge of previous layers. To deal with this problem, we propose PEFNet, a novel model using modified UNet with a new Positional Embedding Feature block in the merging stage, which has more accuracy and generalization in polyps segmentation. The PEF block utilizes the information of the position, concatenated features, and extracted features to enrich the gained knowledge and improve the model's comprehension ability. With EfficientNetV2-L as the backbone, we obtain the IOU score of 0.8201 and the Dice coefficient of 0.8802 on the Kvasir-SEG dataset. By PEFNet, we also take second place on the task Medico: Transparency in Medical Image Segmentation at MediaEval 2021, which is clear proof of the effectiveness of our models.

Authors: Luo, Wei; Xu, Mengying; Lai, Hanjiang

Abstract:

(MSA) aims at recognizing emotion categories by textual, visual, and acoustic cues. However, in real-life scenarios, one or two modalities may be missing due to various reasons. And when text modality is missing, obvious deterioration will be observed since text modality contains much more semantic information compared to vision and audio modality. To this end, we propose the Multimodal Reconstruct and Align Net (MRAN) to tackle the missing modality problem, especially to relieve the decline caused by the text modality's absence. We first propose the Multimodal Embedding and Missing Index Embedding to guide the reconstruction of missing modalities features. Then, visual and acoustic features are projected to the textual feature space, and all three modalities' features are learned to be close to the word embedding of their corresponding emotion category, making visual and acoustic features aligned with textual features. In this text-centered way, vision and audio modality benefit from the more informative text modality. Thus it improves the robustness of the network for different modality missing conditions, especially when text modality is missing. Experimental results conducted on two multimodal benchmarks IEMOCAP and CMU-MOSEI show that our method outperforms baseline methods, gaining superior results on different kinds of modality missing conditions.

Authors: Gao, Wenhua; Zhang, Lanju; Yang, Hao; Zhang, Yuan; Yan, Jinyao; Lin, Tao

Abstract:

With the development of multimedia technology and the upgrading of mobile terminal equipment, short video platforms and applications are becoming more and more popular. Compared with traditional long video, short video users tend to slide from current viewing video more frequently. Unviewed preloaded video chunks cause a large amount of bandwidth waste and do not contribute to improving the user QoE. Since bandwidth savings conflict with user QoE improvements, it is very challenging to satisfy both. To solve this problem, this paper proposes DHP, a joint video download and dynamic bitrate adaptation algorithm for short video streaming. DHP makes the chunk download decision based on the maximum buffer model and retention rate, and makes the dynamic bitrate adaptation decision according to past bandwidth and buffer size. Experimental results show that DHP can reduce the bandwidth waste by at most 66.74% and improve the QoE by at most 42.5% compared to existing solutions under various network conditions.

Authors: Chen, Weiyan; Zhu, Changjian; Zhang, Shan; Xiang, Sen

Abstract:

Occlusion is a common phenomenon in actual scenes, this phenomenon will seriously influence application of light field rendering (LFR) technology. We propose an occlusion model of scene surface that approximating the scene surface as a set of concave and convex parabolas to solve the light field (LF) reconstruction problem. The first step in this model involves determining the occlusion function. After obtaining the occlusion function, we can then perform the plenoptic spectral analysis. Through the plenoptic spectral analysis, the plenoptic spectrum will reveal occlusion characteristics. Finally, the occlusion characteristics can be used to determine the minimal sampling rate and a new reconstruction filter can also be applied to calibrate the aliasing spectrum to achieve high quality view synthesis. This extends previous works of LF reconstruction that considering reconstruction filter. Experimental evaluation demonstrates that our occlusion model significantly address occlusion problem while improve the rendering quality of light field.

Authors: Lorenzi, Daniele; Tashtarian, Farzad; Amirpour, Hadi; Timmerer, Christian; Hellwagner, Hermann

Abstract:

HTTP Adaptive Streaming (HAS) is the predominant technique to deliver video contents across the Internet with the increasing demand of its applications. With the evolution of videos to deliver more immersive experiences, such as their evolution in resolution and framerate, highly efficient video compression schemes are required to ease the burden on the delivery process. While AVC/H.264 still holds the billboard of the most adopted codecs, we are experiencing an increase in the usage of new generation codecs (HEVC/H.265, VP9, AV1, VVC/H.266, etc.). Compared to AVC/H.264, these codecs can either achieve the same quality besides a bitrate reduction or improve the quality while targeting the same bitrate. In this paper, we propose a Mixed-Binary Linear Programming (MBLP) model called Multi-Codec Optimization Model at the edge for Live streaming (MCOM-Live) to jointly optimize (i) the overall streaming costs, and (ii) the visual quality of the content played out by the end-users by efficiently enabling multi-codec content delivery. Given a video content encoded with multiple codecs according to a fixed bitrate ladder, the model will choose among three available policies, i.e., fetch, transcode, or skip, the best option to handle the representations. We compare the proposed model with traditional approaches used in the industry. The experimental results show that our proposed method can reduce the additional latency by up to 23% and the streaming costs by up to 78%, besides improving the visual quality of the delivered segments by up to 0.5 dB, in terms of PSNR.

Authors: Feng, Ping; Zhang, Hanyun; Sun, Yingying; Tang, Zhenjun

Abstract:

This paper proposes a lightweight image hashing based on knowledge distillation and optimal transport for face retrieval. A key contribution is the attention-based triplet knowledge distillation, whose loss function includes attention loss, Kullback-Leibler (KL) loss and identity loss. It can significantly reduce network size with almost no decrease of retrieval performance. Another contribution is the hash quantization based on optimal transport. It partitions the face feature space by calculating class-centers and conducts binary quantization based on the optimal transport. It can make performance improvement in face retrieval with short bit length. In addition, an alternating training strategy is designed for tuning network parameters of our lightweight hashing. Many experiments on two face datasets are carried out to test performances of the proposed lightweight hashing. Retrieval comparisons illustrate that the proposed lightweight hashing outperforms some well-known hashing methods.

Authors: pan, ting; wang, fei; xie, junzhou; liu, weifeng

Abstract:

In order to facilitate the human painting process, numerous research efforts have been made on teaching machines how to "paint like a human", which is a challenging problem. Recent stroke-based rendering algorithms generate non-photorealistic imagery using a number of strokes to mimic a target image. However, the applicability of previous methods can only draw the content of one target image on a canvas that limits generation ability. We propose a novel painting approach that teach machines to paint with multiple target images and then generate new paintings. We consider the order of human painting and propose a combined stroke rendering method that can merge the content of multiple images into the same painting. We use semantic segmentation to obtain semantic information in multiple images, and add the semantic information in different images to the same painting process. Finally, our model can generate new paintings with contents from different images by the guidance of this semantic information. Experimental results demonstrate that our model can effectively generate new paintings which can assist human beings to create.

Authors: Feng, Tianxing; Zhang, Zhe; Xiong, Kaiqiang; Wang, Ronggang

Abstract:

Depth map based Multi-view stereo (MVS) is a task that focuses on taking images from multiple views of one same scene as input, estimating depth in each view, and generating 3D reconstructions of objects in the scene. Though most matching based MVS methods take features of the input images into account, few of them make the best of the underlying global information in images. They may suffer from difficult image regions, such as object boundaries, low-texture areas, and reflective surfaces. Human beings perceive these cases with the help of global awareness, that is to say, the context of the objects we observe. Similarly, we propose Context-guided Multi-view Stereo (ContextMVS), a coarse-to-fine pyramidal MVS network, which explicitly utilizes the context guidance in asymmetrical features to integrate global information into the 3D cost volume for feature matching. Also, with a low computational overhead, we adopt a depth back-projection refined up-sampling module to improve the non-parametric depth up-sampling between pyramid levels. Experimental results indicate that our method outperforms classical learning-based methods by a large margin on public benchmarks, DTU and Tanks and Temples, demonstrating the effectiveness of our method.

Authors: Guo, Jinxin; Zhang, Jiaqiang; Zhang, Xiaojing; Ma, Ming

Abstract:

Action recognition is a crucial task in computer vision and video analysis. The Two-stream network and 3D ConvNets are representative works. Although both of them have achieved outstanding performance, the optical flow and 3D convolution require huge computational effort, without taking into account the need for real-time applications. Current work extracts motion vectors and residuals directly from the compressed video to replace optical flow. However, due to the noisy and inaccurate representation of the motion, the accuracy of the model is significantly decreased when using motion vectors as input. Besides the current works focus only on improving accuracy or reducing computational cost, without exploring the tradeoff strategy between them. In this paper, we propose a light and efficient multi-stream framework, including a motion time fusion module (MTFM) and a double compressed knowledge distillation module (DCKD). MTFM improves the network's ability to extract complete motion information and compensates to some extent for the problem of inaccurate description of motion information by motion vectors in compressed video. DCKD allows the student network to gain more knowledge from teacher with less parameters and input frames. Experimental results on the two public benchmarks(UCF-101 and HMDB-51) outperform the state of the art on the compressed domain.

Authors: Zhao, Shengwei; Liu, Yuying; Du, Shaoyi; Tian, Zhiqiang; Qu, Ting; Xu, Linhai

Abstract:

As a fundamental task in the multimodal domain, text-tovideo retrieval task has received great attention in recent years. Most of the current research focuses on the interaction between cross-modal coarse-grained features. However, the feature granularity of retrieval models has not been fully explored. Therefore, we introduce video internal region information into cross-modal retrieval and propose a cross-model fine-grained feature retrieval framework. Videos are represented as video-frame-region triple features, texts are represented as sentence-word dual features, and the cross-similarity between visual features and text features is computed through token-wise interaction. It effectively extracts the detailed information in the video, guides the model to pay attention to the effective video region information and keyword information in the sentence, and reduces the adverse effects of redundant words and interfering frames. On the most popular retrieval dataset MSRVTT, the framework achieves state-of-the-art results (51.1@1). Excellent experimental results demonstrate the superiority of fine-grained feature interaction.

Authors: Siopi, Maria; Kordopatis-Zilos, Giorgos; Charitidis, Polychronis; Kompatsiaris, Ioannis; Papadopoulos, Symeon

Abstract:

In this paper, we address the problem of image splicing localization with a multi-stream network architecture that processes the raw RGB image in parallel with other handcrafted forensic signals. Unlike previous methods that either use only the RGB images or stack several signals in a channel-wise manner, we propose an encoder-decoder architecture that consists of multiple encoder streams. Each stream is fed with either the tampered image or handcrafted signals and processes them separately to capture relevant information from each one independently. Finally, the extracted features from the multiple streams are fused in the bottleneck of the architecture and propagated to the decoder network that generates the output localization map. We experiment with two handcrafted algorithms, i.e., DCT and Splicebuster. Our proposed approach is benchmarked on three public forensics datasets, demonstrating very competitive performance against several competing methods and achieving state-of-the-art results, e.g., on CASIA with 0.898 AUC.

Authors: Wang, Wei; Lu, Peng; Peng, Xujun; Yin, Wang; Zhao, Zhaoran

Abstract:

The convolutional neural network (CNN) is an effective model for vanishing point (VP) detection, but its success heavily relies on a massive amount of training data to ensure high accuracy. Without sufficient and balanced training data, the obtained CNN-based VP detection models can be easily overfitted with less generalization. By acknowledging that a VP in the image is the intersection of projections of multiple parallel lines in the scene and treating this knowledge as a geometric prior, we propose a prior-guided residual line-shaped convolutional network for VP detection to reduce the dependence of CNN on training data. In the proposed end-to-end approach, the probabilities of VP in the image are computed through an edge extraction subnetwork and a VP prediction subnetwork, which explicitly establishes the geometric relationships among edges, lines, and vanishing points by stacking the differentiable residual line-shaped convolutional modules. Our extensive experiments on various datasets show that the proposed VP detection network improves accuracy and outperforms previous methods in terms of both inference speed and generalization performance.

Authors: Li, Yunhong; Li, Shuai; Yu, Zhenhua

Abstract:

Neural architecture search has attracted much attention because it can automatically find architectures with high performance. In recent years, differentiable architecture search further emerges as one of the main techniques for automatic network design. However, related methods suffer from performance collapse due to excessive skip-connect operations and discretization gaps in search and evaluation. To relieve performance collapse, we propose a polarization regularizer on instance-complexity weighted architecture parameters to push the probability of the most important operation in each edge to 1 while the probabilities of other operations to 0. The polarization regularizer effectively removes the discretization gaps between the search and evaluation procedures, and instance-complexity aware learning of the architecture parameters gives higher weights to hard inputs therefore further improves the network performance. Similar to existing methods, the search process is conducted under a differentiable way. Extensive experiments on a variety of search spaces and datasets show our method can well polarize the architecture parameters and greatly reduce the number of skip-connect operations, which contributes to the performance elevation of network search.

Authors: Liu, Xiaoqiong; Lin, Yuewei; Yang, Qing; Fan, Heng

Abstract:

3D point cloud object tracking has recently witnessed considerable progress relying on deep learning. Such progress, however, mainly focuses on improving tracking accuracy. The risk, especially considering that deep neural network is vulnerable to adversarial perturbations, of a tracker being attacked is often neglected and rarely explored. In order to attract attentions to this potential risk and facilitate the study of robustness in point cloud tracking, we introduce a novel transferable attack network (TAN) to deceive 3D point cloud tracking. Specifically, TAN consists of a 3D adversarial generator, which is trained with a carefully designed multi-fold drift (MFD) loss. The MFD loss considers three common grounds, including classification, intermediate feature and angle drifts, across different 3D point cloud tracking frameworks for perturbation generation, leading to high transferability of TAN for attack. In our extensive experiments, we demonstrate the proposed TAN is able to not only drastically degrade the victim 3D point cloud tracker, i.e., P2B [20], but also effectively deceive other unseen state-of-the-art approaches such as BAT [33] and M2Track [34], posing a new threat to 3D point cloud tracking. Code will be released upon publication.

Authors: Oikonomidis, Alexandros; Pegia, Maria; Moumtzidou, Anastasia; Gialampoukidis, Ilias; Vrochidis, Stefanos; Kompatsiaris, Ioannis

Abstract:

Deep learning methods are widely used in the domain of change detection in remote sensing images. While datasets of that kind are abundant, annotated images, specific for the task at hand, are still scarce. Neural networks trained with Self supervised learning aim to harness large volumes of unlabeled satellite high resolution images to help in finding better solutions for the change detection problem. In this paper we experiment with this approach by presenting 4 different change detection methodologies. We propose a fusion method that under specific parameters can provide better results. We evaluate our results using two openly available datasets with Sentinel-2 satellite images, S2MTCP and OSCD, and we investigate the impact of using 2 different Sentinel 2 band combinations on our final predictions. Finally we conclude by summarizing the benefits of this approach as well as we propose future areas of interest that could be of value in enhancing the change detection task's outcomes.

Authors: Ouyang, Jiajun; Lv, Qingxuan; Zhang, Shu; Dong, Junyu

Abstract:

The main goal of unsupervised domain adaptation is to improve the classification performance on unlabeled data in target domains. Many methods try to reduce the domain gap by treating multiple domains as one to enhance the generalization of a model. However, aligning domains as a whole does not account for instance-level alignment, which might lead to sub-optimal results. Currently, many researchers utilize meta-learning and instance segmentation approaches to tackle this problem. But it can only obtain a further optimized the domain-invariant feature learned by the model, rather than achieve instance-level alignment. In this paper, we interpret unsupervised domain adaptation from a new perspective, which exploits the energy difference between the source and target domains to reduce the performance drops caused by the domain gap. At the same time, we improve and exploit the contrastive learning loss, which can push the target domain away from the decision boundary. The experimental results on different benchmarks against a range of the state-of-the-art approaches justify the performance and the effectiveness of the proposed method.

Authors: Chen, Song; Wang, Chong; Liu, Weijie; Ye, Zhengjie; Deng, Jiacheng

Abstract:

Few-Shot Object Detection (FSOD) task is widely used in various data-scarce scenarios, aiming to expand the object detector with a few novel class samples. The current mainstream FSOD models improve the accuracy by mining novel class instances in the training set and fine-tuning the detector with mined pseudo-label data. Substantial progress has been made using pseudo-label ap-proaches, but the impact of pseudo-label diversity on FSOD tasks has not been explored. In our work, for the purpose of fully utilizing the pseudo-label data and exploring their diversity, we propose a new framework mainly including Novel Instance Bank (NIB) and Correlation-Guided Loss Correction (CGLC). Dynamically updated NIB stores the novel class instances to ensure the diver-sity of novel instances in each batch. Moreover, to better exploit the pseudo-label diversity, CGLC adaptively employs k-shot samples to guide correct and incorrect pseudo-labels to pull away from each other. Experimental results on the MS-COCO dataset demonstrate the effectiveness of our method, which does not require any additional training samples or parameters. Our code is available at: https://github.com/lotuser1/PDE.

Authors: Gan, Xiangqi; Zhu, Changjian; Bai, Mengqin; Wei, Ying; Chen, Weiyan

Abstract:

Light field rendering technology is an important tool, which is applied a set of multi view image to render realistic novel views and experiences through some simple interpolation. However, the rendered novel views often have various distortions or low quality due to complexity of the scene, e.g., occlusion and non Lambertian. The distortion of novel views in the spectrum of light field signal can be reflected in periodic aliasing. In this paper, we propose a spectrum dependent depth layered (SDDL) model to eliminate the spectrum aliasing of light field signals, so as to improve the rendering quality of novel views. The SDDL model is about taking advantage of the characteristics that light field signal spectrum structure is only limited by the minimum depth and the maximum depth. So we in manner of increasing the depth of the layer between the minimum and maximum depth,it will reduce the sampling interval between cameras, and the adjacent two sampling during the period of the spectrum interval will become bigger. Thus, the aliasing of novel views will become smaller, it can be eliminated by this method to achieve the purpose of aliasing. In fact,the result of experiment prove our method can improve the rendering quality of light field.

Authors: Shinoda, Kazutoshi; Takezawa, Yuki; Suzuki, Masahiro; Iwasawa, Yusuke; Matsuo, Yutaka

Abstract:

An interactive instruction following task has been proposed as a benchmark for learning to map natural language instructions and first-person vision into sequences of actions to interact with objects in 3D environments. We found that an existing end-to-end neural model for this task tends to fail to interact with objects of unseen attributes and follow various instructions. We assume that this problem is caused by the high sensitivity of neural feature extraction to small changes in vision and language inputs. To mitigate this problem, we propose a neuro-symbolic approach that utilizes high-level symbolic features, which are robust to small changes in raw inputs, as intermediate representations. We verify the effectiveness of our model with the subtask evaluation on the ALFRED benchmark. Our experiments show that our approach significantly outperforms the end-to-end neural model by 9, 46, and 74 points in the success rate on the ToggleObject, PickupObject, and SliceObject subtasks in unseen environments respectively.

Authors: Deng, Xuran; Liu, Chuanbin; Lu, Zhiying

Abstract:

Fine-grained visual categorization (FGVC) is a challenging task in the image analysis field which requires comprehensive discriminative feature extraction and representation. To get around this problem, previous works focus on designing complex modules, the so-called necks and heads, over simple backbones, while bringing a huge computational burden. In this paper, we bring a new insight: Vision Transformer itself is an all-in-one FGVC framework that consists of basic Backbone for feature extraction, Neck for further feature enhancement and Head for selecting discriminative feature. We delve into the feature extraction and representation pattern of ViT for FGVC and empirically show that simply recombining the original ViT structure to leverage multi-level semantic representation without introducing any other parameters is able to achieve higher performance. Under such insight, we proposed RecViT, a simple recombination and modification of original ViT, which can capture multi-level semantic features and facilitate fine-grained recognition. In RecViT, the deep layers of the original ViT are served as Head, a few middle layers as Neck and shallow layers as Backbone. In addition, we adopt an optional Feature Processing Module to enhance discriminative feature representation at each semantic level and align them for final recognition. With the above simple modifications, RecViT obtains significant improvement in accuracy in FGVC benchmarks: CUB-200-2011, Stanford Cars and Stanford Dogs.

Authors: Xie, Xinjia; Liu, Feng; Gai, Shun; Huang, Zhen; Hu, Minghao; Wang, Ankun

Abstract:

Dense Retriever (DR) for Open-domain textual question answering (OpenQA), which aims to retrieve passages from large data sources like Wikipedia or Google, has gained wide attention in recent years. Although DR models continuously refresh state-of-the-art performances, their improvement relies on negative sampling during the training process. Existing sampling strategies mainly focus on developing a complex algorithm based on computer science, and ignore the abundant semantic features of datasets. We discover that there exists obvious changes in semantic similarity and present a three-level hierarchy of semantic similarity: same topic, same class, other class, whose rationality is further demonstrated by ablation study. Based on this, we propose a hard negative sampling strategy named Hierarchical Semantic Similarity (HSS). Our HISSIM model performs negative sampling at semantic levels of topic and class, and experimental results on four datasets show that it achieves comparable or better retrieval performance compared with existing competitive baselines. The code is available in https://github.com/redirecttttt/HSS.

Authors: Yang, Jing; Chen, Junwen; Yanai, Keiji

Abstract:

mutual retrieval between recipe images and texts, which is clear for human but arduous to formulate. Although many previous works endeavored to solve this problem, most works did not efficiently exploit the cross-modal information among recipe data. In this paper, we present a frustratingly straightforward cross-modal recipe retrieval framework, Transformer-based Network for Large Batch Training (TNLBT) achieving high performance on both recipe retrieval and image generation tasks, which is designed to efficiently exploit the rich cross-modal information. In our proposed framework, Transformer-based encoders are applied for both image and text encoding for cross-modal embedding learning. Furthermore, we apply generative adversarial network to force learned recipe embedding to recover visual information to encourage models to retain modality-specific information. We also adopt several loss functions like selfsupervised learning loss on recipe text to encourage the model to further promote the cross-modal embedding learning. Since contrastive learning could benefit from a larger batch size according to the recent literature on self-supervised learning, we also adopt a large batch size during training and have validated its effectiveness. In the experiments, the proposed framework significantly outperformed the current state-of-the-art frameworks in both cross-modal recipe retrieval and image generation tasks on the benchmark Recipe1M by a huge margin. We also found that CLIP-ViT performs better than ViT-B as the image encoder backbone in the experiment. This is the first work which confirmed the effectiveness of large batch training on cross-modal recipe embedding learning.

Authors: Lin, Jiaqin; Du, Shaoyi; Liu, Yuying; Tian, Zhiqiang; Qu, Ting; Zheng, Nanning

Abstract:

Traffic accidents caused by driver fatigue lead to millions of death and financial loss every year. Current end-to-end methods for driver fatigue detection are not capable of distinguishing the detailed fatigue symptoms and interpretably inferring the fatigue state. In this paper, we propose an interpretable driver fatigue detection method with hierarchical fatigue symptom representations. In pursuit of a more general and interpretable driver fatigue detection approach, we propose to detect detailed fatigue symptoms before driver state inferring. First of all, we propose a hierarchical method that accurately classifies abnormal behaviors into detailed fatigue symptoms. Moreover, to fuse the fatigue symptom detection results accurately and efficiently, we propose an effective and interpretable fatigue estimation method with maximum a posteriori and experience constraints. Finally, we evaluate the proposed method on the driver fatigue detection benchmark dataset, and the experimental results endorse the feasibility and effectiveness of the proposed method.

Authors: Gao, Ming; Wu, Shilian; Wang, Zengfu

Abstract:

Due to the widespread use of English, considerable attention has been paid to scene text recognition with English as the target language, rather than multilingual scene text recognition. However, it is increasingly necessary to recognize multilingual texts with the continuous advancement of global integration. In this paper, a Length-sensitive Language-bound Recognition Network(LLRN) is proposed for multilingual text recognition. LLRN follows the traditional encoder-decoder structure. We improve the encoder and decoder respectively to better adapt to multilingual text recognition. On the one hand, we propose a Length-sensitive Encoder(LE) to encode features of different scales for long-text images and short-text images respectively. On the other hand, we present a Language-bound Decoder(LD). LD leverages language prior information to constrain the original output of the decoder to further modify the recognition results. Moreover, to solve the problem of multilingual data imbalance, we propose a Language-balanced Data Augmentation(LDA) approach. Experiments show that our method outperforms English-oriented mainstream models and achieves state-of-the-art results on MLT-2019 multilingual recognition benchmark.

Authors: Fang, Jingsen; Shi, Shoudong; Fang, Yi; Huo, Zheng

Abstract:

It is difficult to maintain a standard sitting posture for long periods of time, and a non-standard sitting posture can damage human health. Therefore, it is important to detect sitting posture in real time and remind users to adjust to a healthy sitting posture. Deep learning-based sitting posture recognition methods currently achieve better improvement in recognition accuracy, but the models cannot combine high accuracy and speed on embedded platforms, which in turn makes it difficult to be applied in edge intelligence. Aiming to overcome the challenge, we propose a fast sitting posture recognition method based on OpenPose, using a ShuffleNetV2 network to replace the original backbone network to extract the underlying features, and using a cosine information distance to find redundant filters to prune and optimize the model. Lightweight model after pruning for more efficient real-time interaction. At the same time, the sitting posture recognition method is improved by fusing joint distance and angle features on the basis of skeletal joint features to improve the accuracy while ensuring the recognition detection speed. The optimized model can not only run at 8 fps on the Jetson Nano embedded device, but can also ensure recognition accuracy of 94.73%. Experimental results show that the improved model can meet the real-time detection of sitting posture on embedded devices.

Authors: Yin, Yuanhang; Hua, Yang; Song, Tao; Ma, Ruhui; Guan, Haibin

Abstract:

Multi-object tracking is a challenging video task that requires both locating the objects in the frames and associating the objects among the frames, which usually utilizes the tracking-by-detection paradigm. Supervised multi-object tracking methods have made stunning progress recently, however, the expensive annotation costs for bounding boxes and track ID labels limit the robustness and generalization ability of these models. In this paper, we learn a novel multi-object tracker using only unlabeled videos by designing a self-supervisory learning signal for an association model. Specifically, inspired by the cycle-consistency used in video correspondence learning, we propose to track the objects forwards and backwards, i.e., each detection in the first frame is supposed to be matched with itself after the forward-backward tracking. We utilize this cycle-consistency as the self-supervisory learning signal for our proposed multi-object tracker. Experiments conducted on the MOT17 dataset show that our model is effective in extracting discriminative association features, and our tracker achieves competitive performance compared to other trackers using the same pre-generated detections, including UNS20, Tracktor++, FAMNet, and CenterTrack.

Authors: Tran, Ly-Duyen; Nie, Dongyun; Gurrin, Cathal; Nguyen, Binh; Zhou, Liting

Abstract:

Organising and preprocessing are crucial steps in order to perform analysis on lifelogs. This paper presents a method for preprocessing, enriching, and segmenting lifelogs based on GPS trajectories and images captured from wearable cameras. The proposed method consists of four components: data cleaning, stop/trip point classification, post-processing, and event characterisation. The novelty of this paper lies in the incorporation of a visual module (using a pretrained CLIP model) to improve outlier detection, correct classification errors, and identify each event's movement mode or location name. This visual component is capable of addressing imprecise boundaries in GPS trajectories and the partition of clusters due to data drift. The results are encouraging, which further emphasises the importance of visual analytics for organising lifelog data.

Authors: Zhang, Yuan; Tian, Xiang; Zhang, Ziyang; Xu, Xiangmin

Abstract:

The increasing capability of networks for facial expression recognition with disturbing factors is often accompanied by a large computational burden, which imposes limitations on practical applications. In this paper, we propose a lightweight multi-level information fusion network with distillation loss, which can be more lightweight compared with other methods under the premise of not losing accuracy. The multi-level information fusion block uses fewer parameters to focus on information from multiple levels with greater detail awareness, and the channel attention used in this block allows the network to concentrate more on sensitive information when processing facial images with disturbing factors. In addition, the distillation loss makes the network less susceptible to the errors of the teacher network. The proposed method has the fewest parameters of 0.98 million and GFLOPs of 0.142 compared with the state-of-the-art methods while achieving 88.95%, 64.77%, 60.63%, and 62.28% on the datasets RAF-DB, AffectNet-7, AffectNet-8, and SFEW, respectively. Abundantly experimental results show the effectiveness of the method.

Authors: Loupas, Georgios; Pistola, Theodora; Diplaris, Sotiris; Ioannidis, Konstantinos; Vrochidis, Stefanos; Kompatsiaris, Ioannis

Abstract:

Folk dances consist an important part of the Intangible Cultural Heritage (ICH) of each place. Nowadays, there is a great amount of videos related to folk dances. An automatic dance recognition algorithm can ease the management of this content and enforce the promotion of folk dances to the younger generations. Automatic dance recognition is still an open research area that belongs to the more general field of human activity recognition. Our work focuses on the exploration of existing deep neural network architectures for automatic recognition of Greek folk dances depicted in standard videos, as well as the experimentation with different representations of input. For our experiments, we have collected YouTube videos of Greek folk dances from north-eastern Greece. Specifically, we have validated three different deep neural network architectures using raw RGB and grayscale video frames, optical flow, as well as “visualised” multi-person 2D poses. In this paper, we describe our experiments, and, finally, we present the results and findings of the conducted research.

Authors: Lin, Chih-Wei; Chen, Zhongsheng; Huang, Xiuping; Yang, Suhui

Abstract:

This paper proposes the dual-dimension and dual-scale spatiotemporal convolutional neural network, namely DDS-CNN, which consists of two modules, the global spatiotemporal module (GSM) and the local spatiotemporal module (LSM), for precipitation intensity recognition. The GSM uses 3D LSTM operations to study the influence of the relationship between sampling points on precipitation. The LSM takes 4D convolution operations and forms the convolution branches with various convolution kernels to learn the rain pattern of different precipitation. We evaluate the performance of DDS-CNN using the self-collected dataset, IMLab-RAIN-2018, and compare it with the state-of-the-art 3D models. DDS-CNN has the highest overall accuracy and achieves 98.63%. Moreover, we execute the ablation experiments to prove the effectiveness of the proposed modules.

Authors: Zhao, Xin; Ren, Zhihang

Abstract:

Skin lesions segmentation from dermoscopic images has been a long-standing challenging problem, which is important for improving the analysis of skin cancer. Due to the large variation of melanin in the lesion area, the large number of hairs covering the lesion area, and the unclear boundary of the lesion, most previous works were hard to accurately segment the lesion area. In this paper, we propose a Multi-Scale Gaussian Difference Preprocessing and Dual Stream CNN-Transformer Hybrid Network for Skin Lesion Segmentation, which can accurately segment a high-fidelity lesion area from a dermoscopic image. Specifically, we design three specific sets of Gaussian difference convolution kernels to significantly enhance the lesion area and its edge information, conservatively enhance the lesion area and its edge information, and remove noise features such as hair. Through the information enhancement of multi-scale Gaussian convolution, the model can easily extract and represent the enhanced lesion information and lesion edge information while reducing the noise information. Secondly, we adopt dual steam network to extract features from the Gaussian difference image and the original image separately and fuse them in the feature space to accurately align the feature information. Thirdly, we apply the convolution neural network (CNN) and vision transfomer (ViT) hybrid architectures to better exploit the local and global information. Finally, we use the coordinate attention mechanism and the self-attention mechanism to enhance the sensitivity to the necessary features. Extensive experimental results on the ISIC 2016, PH2, and ISIC 2018 dataset demonstrate that our proposed approach achieves compelling performance in skin lesions segmentation.

Authors: Dang-Nguyen, Duc-Tien; Sjøen, Vegard Velle; Le, Dinh-Hai; Dao, Thien-Phu; Tran, Anh-Duy; Tran, Minh-Triet

Abstract:

The research done in this study has delved deeply into the changes made to digital images that are uploaded to three of the major social media platforms and image storage services in today's society: Facebook, Flickr, and Google Photos. In addition to providing up-to-date data on an ever-changing landscape of different social media networks' digital fingerprints, a deep analysis of the social networks' filename conventions has resulted in two new approaches in (i) estimating the true upload date of Flickr photos, regardless of whether the dates have been changed by the user or not, and regardless of whether the image is available to the public or has been deleted from the platform; (ii) revealing the photo ID of a photo uploaded to Facebook based solely on the file name of the photo.

Authors: Fu, Yingnan; Zheng, Shu; Cai, Wenyuan; Gao, Ming; Jin, Cheqing; Zhou, Aoying

Abstract:

Structural image content recognition (SICR) aims to transcribe a two-dimensional structural image (e.g., mathematical expression, chemical formula, or music score) into a token sequence. Existing methods are mainly encoder-decoder based and overlook the importance of feature selection and spatial relation extraction in the feature map. In this paper, we propose DEAL (shorted for Dynamic fEAture seLection) for SICR, which contains a dynamic feature selector and a spatial relation extractor as two cornerstone modules. Specifically, we propose a novel loss function and random exploration strategy to dynamically select useful image cells for target sequence generation. Further, we consider the positional and surrounding information of cells in the feature map to extract spatial relations. We conduct extensive experiments to evaluate the performance of DEAL. Experimental results show that DEAL outperforms other state-of-the-arts significantly.

Authors: Li, Lei; Lai, Hong

Abstract:

Tensor networks is a numerical tool in quantum many-body systems, it have been widely used in machine learning. The projected entangled pair states (PEPS) tensor network with a similar structure to the image achieves significant superiority compared to other tensor network models in the image classification task. Based on the PEPS tensor network, this paper constructs a Multi-layered PEPS (MLPEPS) tensor network model for supervised learning. PEPS is used to extract features layer by layer from the image mapped to the Hilbert space, which fully utilizes the correlation between pixels while retaining the global structural information of the image. By performing classification tasks on Fashion-MNIST and COVID-19 Radiography datasets, MLPEPS outperforms other tensor network models including PEPS. Under the same experimental conditions, the learning ability of MLPEPS is close to that of some well-known neural networks. MLPEPS can obtain different structures by setting different number of layers and PEPS block size, so it has great potential in machine learning.

Authors: Dong, Peijie; Niu, Xin; Pan, Hengyue; Li, Dongsheng; huang, Zhen

Abstract:

The search space is crucial in neural architecture search (NAS), and can determine the upper limit of the performance. Most methods focus on the design of depth and width when designing the search space, ignoring the receptive field. With a larger receptive field, the model is able to aggregate hierarchical information and strengthen its representational power. However, expanding the receptive fields directly with large convolution kernels suffers from high computational complexity. we instead enlarge the receptive field by introducing pooling operations with little overhead. In this paper, we propose a method named Auto Learning Receptive Fields (AutoRF), which is the first attempt at the auto attention module design with regard to the adaptive receptive field. In this paper, we present a pooling-based auto-learning approach for receptive field search. Our proposed search space encompasses typical multi-scale receptive field integration modules theoretically. Detailed experiments demonstrate the generalization ability of AutoRF and outperform various hand-crafted methods as well as NAS-based ones.

Authors: Batziou, Elissavet; Ioannidis, Konstantinos; Patras, Ioannis; Vrochidis, Stefanos; Kompatsiaris, Ioannis

Abstract:

The inevitable environmental and technical limitations of image capturing has as a consequence that many images are frequently taken in inadequate and unbalanced lighting conditions. Low-light image enhancement has been very popular for improving the visual quality of image representations, while low-light images often require advanced techniques to improve the perception of information for a human viewer. One of the main objectives in increasing the lighting conditions is to retain patterns, texture, and style with minimal deviations from the considered image. To this direction, we propose a low-light image enhancement method with Haar wavelet-based pooling to preserve texture regions and increase their quality. The presented framework is based on the U-Net architecture to retain spatial information, with a multi-layer feature aggregation (MFA) method. The method obtains the details from the low-level layers in the stylization processing. The encoder is based on dense blocks, while the decoder is the reverse of the encoder, and extracts features that reconstruct the image. Experimental results show that the combination of the U-Net architecture with dense blocks and the wavelet-based pooling mechanism comprises an efficient approach in low-light image enhancement applications. Qualitative and quantitative evaluation demonstrates that the proposed framework reaches state-of-the-art accuracy but with less resources than LeGAN.

Authors: Wu, Zhihong; Qu, Xiwen; Huang, Jun; Wu, Xuangou

Abstract:

In-air handwriting is a new and more humanized way of human-computer interaction, which has a broad application prospect. One of the existing online handwritten Chinese text recognition model is to convert the trajectory data into image-like representation and use two-dimensional convolutional neural network (2DCNN) for feature extraction, and the another one directly process trajectory sequence with Long Short-Term Memory (LSTM). However, when using 2DCNN, many information will be lost in the process of conversion into images. When using LSTM, LSTM network is easy to cause gradient problem. So we propose an attention convolutional recurrent network (ACRN) for in-air handwritten Chinese text, which introduces one-dimensional convolutional neural network (1DCNN) containing dilation convolution for feature extraction of trajectory data directly. After that, the ACRN uses LSTM combined with multihead attention mechanism to focus on some key words in handwritten Chinese text, mines multi-level dependencies and outputs to softmax for classification. Finally the ACRN uses the Connectionist Temporal Classification (CTC) objective function without input-output alignment to decode the coding results. We conduct experiments on the CASIA-OLHWDB2.0-2.2 dataset and in-air handwritten Chinese text dataset IAHCT-UCAS2018. Experimental results demonstrate that compared with previous methods, our method obtains a more compact model with a higher recognition accuracy.