Lianyu Hu is a 4th-year PhD candidate in Tianjin University, China, supervised by Prof Wei Feng. During his graduate student period, he worked closely with Prof Shenglan Liu. His research interest includes Video Understanding, Sign Lnaguage Understanding and Multimodal Learning.

🔥 News

• We release iLLaVA, an efficient method for large vision language models by merging visual tokens. It could achieve about 2× throughput and 1.7× - 2× memory reduction with comparable performance through merging redundant visual tokens in some certain layers.

• We release Deep Correletaed Prompting, which tackles the missing-modality scenarios by proposing three different types of prompting approaches, largely improving the robustness of large vision-language models.

• We release CorrNet+, an unified model with superior performance on both continuous sign language recognition and sign language translation tasks by using only RGB inputs.

• We release DSTA-SLR, which performs sign language recognition (SLR) with pure skeleton inputs but ahcieves comparable accuracy and much faster speed than recognition with RGB inputs.

📝 Publications

📖 PrePrint

• iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models, Lianyu Hu, Fanhua Shang, Liang Wan, Wei Feng. 2024.12. [code].

• CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation, Lianyu Hu, Wei Feng, Liqing Gao, Zekang Liu, Liang Wan. 2024.04. [code].

• Improving Continuous Sign Language Recognition with Adapted Image Models, Lianyu Hu, Tongkai Shi, Liqing Gao, Zekang Liu, Wei Feng. 2024.04. [code].

🖊️ Selected Publications

• Deep Correlated Prompting for Visual Recognition with Missing Modalities. Lianyu Hu, Tongkai Shi, Wei Feng, Fanhua Shang, Liang Wan. NeurIPS 2024. [code].

• Pose-Guided Fine-Grained Sign Language Video Generation. Tongkai Shi, Lianyu Hu, Fanhua Shang, Jichao Feng, Peidong Liu, Wei Feng. ECCV 2024. [code].

• Spatial Temporal Aggregation for Efficient Continuous Sign Language Recognition. Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. IEEE Transactions on Emerging Topics in Computational Intelligence.

• Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition. Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. COLING 2024. [code].

• COMMA: Co-Articulated Multi-Modal Learning. Lianyu Hu, Liqing Gao, Zekang Liu, Chi-Man Pun, Wei Feng. AAAI 2024. [code].

• Scalable Frame Resolution for Efficient Continuous Sign Language Recognition. Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. Pattern Recognition.

• AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition. Lianyu Hu, Liqing Gao, Zekang Liu, Chi-Man Pun, Wei Feng. ACMMM 2023 (Oral). [code].

• Skeleton-Based Action Recognition with Local Dynamic Spatial-Temporal Aggregation. Lianyu Hu, Shenglan Liu, Wei Feng. Expert Systems with Applications. [code]. (Previous name: Spatial Temporal Graph Attention Network for Skeleton-Based Action Recognition)

• Continuous Sign Language Recognition with Correlation Network. Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. CVPR 2023. [code].

• Self-Emphasizing Network for Continuous Sign Language Recognition. Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. AAAI 2023 (Oral). [code].

• Temporal Lift Pooling for Continuous Sign Language Recognition.Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. ECCV 2022. [code].

• HFNet: A Novel Model for Human Focused Sports Action Recognition.Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng. ACMMM 2020 Workshop.

🎖 Honors and Awards

• 2024.12, 优秀学生标兵（ten per year）
• 2024.10, National Scholarship
• 2023.10, National Scholarship

📖 Educations

• 2021-now, PhD candidate in Computer Science and Technology, Tianjin Univerisity
• 2018-2021, MEng in Computer Science and Technology, Dalian University of Technology
• 2014-2018, BSc in Electronics and Information Engineering, Dalian University of Technology